Plants Seedlings Classification
In recent times, the field of agriculture has been in urgent need of modernizing, since the amount of manual work people need to put in to check if plants are growing correctly is still highly extensive. Despite several advances in agricultural technology, people working in the agricultural industry still need to have the ability to sort and recognize different plants and weeds, which takes a lot of time and effort in the long term.
The potential is ripe for this trillion-dollar industry to be greatly impacted by technological innovations that cut down on the requirement for manual labor, and this is where Artificial Intelligence can benefit the workers in this field, as the time and energy required to identify plant seedlings will be greatly shortened by the use of AI and Deep Learning. The ability to do so far more efficiently and even more effectively than experienced manual labor could lead to better crop yields, the freeing up of human involvement for higher-order agricultural decision making, and in the long term will result in more sustainable environmental practices in agriculture as well.
To classify the plant seedlings into their respective categories.
The Aarhus University Signal Processing group, in collaboration with the University of Southern Denmark, has provided the data containing images of unique plants belonging to 12 different species.
There are 12 plant species (categories) in the dataset:
Import Libraries and Basics
#visualkeras is a library to visualize cnn architectures
!pip install visualkeras
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting visualkeras
Downloading visualkeras-0.0.2-py3-none-any.whl (12 kB)
Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from visualkeras) (8.4.0)
Collecting aggdraw>=1.3.11
Downloading aggdraw-1.3.16-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (993 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 993.0/993.0 kB 30.3 MB/s eta 0:00:00
Requirement already satisfied: numpy>=1.18.1 in /usr/local/lib/python3.10/dist-packages (from visualkeras) (1.22.4)
Installing collected packages: aggdraw, visualkeras
Successfully installed aggdraw-1.3.16 visualkeras-0.0.2
#library for creating data paths
import os
#library for randomly selecting data points
import random
#library for performing numerical computations
import numpy as np
#library for data manipulation and analysis
import pandas as pd
#library for data visualization
import seaborn as sns
#library for creating and showing data
import matplotlib.pyplot as plt
#library for reading and showing images
import matplotlib.image as mpimg
#Python Imaging Library to handle image files
from PIL import Image
#OpenCv for image and video processing
import cv2
#sklearn
from sklearn.model_selection import train_test_split
from sklearn.utils import class_weight
from sklearn.utils.class_weight import compute_sample_weight
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
from sklearn.preprocessing import LabelBinarizer
#tensorflow
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.preprocessing.image import img_to_array, load_img
#importing all the required sub-modules from Keras
from keras import backend
from keras.utils import to_categorical
from keras.models import Sequential, Model
from keras.applications.vgg16 import VGG16
from keras.preprocessing.image import ImageDataGenerator
from keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, BatchNormalization, Dropout
from keras import losses, optimizers
from keras.optimizers import RMSprop,Adam,SGD #optimiers for optimizing the model
from keras.callbacks import EarlyStopping, ModelCheckpoint #regularization method to prevent the overfitting
from keras.utils.vis_utils import plot_model
#library to visualize model architecture
import visualkeras
#ignore warnings that may occur due to deprecations
import warnings
warnings.filterwarnings("ignore")
Setup Google Drive
#connect to google drive
from google.colab import drive
drive.mount('/content/drive')
Mounted at /content/drive
#parent directory where images are stored
image_file = '/content/drive/MyDrive/ML/CV/images.npy' #npy file containing images
label_file = '/content/drive/MyDrive/ML/CV/Labels.csv' #csv file containing the labels for each image of the npy file
First we will load the image_files and their corresponding labels.
#load the numpy array
img_array = np.load(image_file)
#visualize contents of array, index 0
img_array[0]
array([[[ 35, 52, 78],
[ 36, 49, 76],
[ 31, 45, 69],
...,
[ 78, 95, 114],
[ 76, 93, 110],
[ 80, 95, 109]],
[[ 33, 46, 68],
[ 37, 50, 73],
[ 48, 65, 83],
...,
[ 81, 96, 113],
[ 74, 89, 105],
[ 83, 95, 109]],
[[ 34, 50, 68],
[ 35, 52, 72],
[ 70, 85, 101],
...,
[ 83, 97, 112],
[ 79, 94, 108],
[ 79, 94, 107]],
...,
[[ 35, 50, 69],
[ 42, 57, 73],
[ 42, 57, 72],
...,
[ 60, 76, 92],
[ 67, 81, 97],
[ 64, 77, 95]],
[[ 36, 52, 67],
[ 48, 63, 78],
[ 41, 57, 73],
...,
[ 44, 66, 83],
[ 58, 76, 91],
[ 57, 74, 90]],
[[ 44, 58, 70],
[ 43, 57, 73],
[ 40, 55, 72],
...,
[ 41, 70, 92],
[ 55, 78, 97],
[ 61, 79, 96]]], dtype=uint8)
Shape of Array images
#shape of the image array
img_array.shape
(4750, 128, 128, 3)
The image array contains 4,750 records. Each record has a length of 128 rows and 128 columns (resolution in pixels). And each element has three values representing Red, Green, Blue (RGB).
#read the labels and store in a dataframe
label_df = pd.read_csv(label_file)
#display the head of the labels dataset
label_df.head(10)
| Label | |
|---|---|
| 0 | Small-flowered Cranesbill |
| 1 | Small-flowered Cranesbill |
| 2 | Small-flowered Cranesbill |
| 3 | Small-flowered Cranesbill |
| 4 | Small-flowered Cranesbill |
| 5 | Small-flowered Cranesbill |
| 6 | Small-flowered Cranesbill |
| 7 | Small-flowered Cranesbill |
| 8 | Small-flowered Cranesbill |
| 9 | Small-flowered Cranesbill |
Shape of dataframe Labels
#print shape of label dataframe
label_df.shape
(4750, 1)
There are 4750 records in total.
#display information of the label dataframe and its data type
label_df['Label'].info
<bound method Series.info of 0 Small-flowered Cranesbill
1 Small-flowered Cranesbill
2 Small-flowered Cranesbill
3 Small-flowered Cranesbill
4 Small-flowered Cranesbill
...
4745 Loose Silky-bent
4746 Loose Silky-bent
4747 Loose Silky-bent
4748 Loose Silky-bent
4749 Loose Silky-bent
Name: Label, Length: 4750, dtype: object>
We should convert 'Label' to a string type because later we will convert it to one-hot vectors. To convert to one-hot vectors, we will use 'to_categorical' which expects an array of integers or numeric values. Otherwise type may throw an error.
#convert to string type
label_df['Label'] = label_df['Label'].astype(str)
Number of Unique Classes
#Number of unique classes (labels)
num_classes = label_df.nunique()[0]
print("The number of seed classes is ",num_classes)
The number of seed classes is 12
Name of Classes
#create an array containing the labels of classes
name_classes = label_df['Label'].unique()
name_classes
array(['Small-flowered Cranesbill', 'Fat Hen', 'Shepherds Purse',
'Common wheat', 'Common Chickweed', 'Charlock', 'Cleavers',
'Scentless Mayweed', 'Sugar beet', 'Maize', 'Black-grass',
'Loose Silky-bent'], dtype=object)
Distribution of Classes
#visualize the seedlings label
figsize=(14, 8)
count_plot = sns.countplot(data=label_df, x="Label")
#rotate x-axis labels
count_plot.set_xticklabels(count_plot.get_xticklabels(), rotation=90)
count_plot.set(xlabel='seed')
plt.show()
#Number of items per categories
label_df.value_counts()
Label Loose Silky-bent 654 Common Chickweed 611 Scentless Mayweed 516 Small-flowered Cranesbill 496 Fat Hen 475 Charlock 390 Sugar beet 385 Cleavers 287 Black-grass 263 Shepherds Purse 231 Common wheat 221 Maize 221 dtype: int64
The data is imbalanced. Class Maize and Common Wheat are roughly 1/3 the size of Loose Silky-bent.
#convert dataframe to array
label_array = label_df.to_numpy().squeeze()
Randomly from the entire set
#Displaying images randomly
#Generate random indices to select rows x columns images
rows = 4
cols = 5
ix_img = np.random.choice(len(label_array), size=rows*cols, replace=False)
# Select the images and labels
images = img_array[ix_img]
labels = label_array[ix_img].tolist()
# Define the size of the figure
fig = plt.figure(figsize=(12, 12))
# Create a rows x cols grid of subplots
for i in range(rows*cols):
ax = fig.add_subplot(rows, cols, i+1)
ax.imshow(images[i])
ax.set_xlabel(str(labels[i]), fontsize=10)
ax.set_xticks([])
ax.set_yticks([])
# Show the figure
plt.tight_layout()
plt.subplots_adjust(wspace=0.2)
plt.show()
Randomly for each class
#Displaying images (randomly) from each category
num_images = 5
# Create a dictionary to store the indices of each category
category_indices = {}
for category in range(num_classes):
category_indices[category] = label_df[label_df['Label'] == category].index.tolist()
# Loop through each unique label
for label in name_classes:
# Get the indices of all images with the current label
indices = np.where(label_df['Label'] == label)[0]
# Choose num_images random indices
random_indices = np.random.choice(indices, size=num_images, replace=False)
# Get the corresponding images from "img_array"
images = img_array[random_indices]
# Create a plot of the images
fig, ax = plt.subplots(1, num_images, figsize=(20, 4))
for i, image in enumerate(images):
ax[i].imshow(image)
ax[i].axis('off')
# Set the suptitle and adjust the spacing
fig.suptitle(label, fontsize=16)
fig.subplots_adjust(top=1.1, bottom=0.1)
plt.show()
Output hidden; open in https://colab.research.google.com to view.
From the dataset image analysis we foresee the need to remove noise from the images. To do this we can use Gaussian Blurring.
We will perform Gaussian Blurring to smooth the images by averaging the intensity of each pixel with the intensity of neighboring pixels.
# Choose a random index for the image
index = np.random.randint(0, img_array.shape[0])
# Extract the image and its label
image = img_array[index, :, :, :]
label = label_df.iloc[index, 0]
fig = plt.figure(figsize=(18, 18))
# Plot the original image
plt.subplot(1, 2, 1)
plt.imshow(image.squeeze(), cmap='gray')
plt.title('Original Image (Label: {})'.format(label))
plt.axis('off')
(-0.5, 127.5, 127.5, -0.5)
# Define the kernel size for the Gaussian blur
kernel_size = (3, 3)
# Apply Gaussian blur to each image in the img_array
for i in range(img_array.shape[0]):
img = img_array[i,:,:,:]
img = cv2.GaussianBlur(img, kernel_size, 0)
img_array[i,:,:,:] = img
After Gaussian Blurring we will scale the pixel values to a fixed range (i.e. 0 to 1).
# Convert the data type of the img_array to float32
img_array = img_array.astype('float32')
# Normalize the pixel values to the range of 0 to 1
img_array = img_array / 255.0
# Plot the preprocessed image
fig = plt.figure(figsize=(18, 18))
plt.subplot(1, 2, 2)
plt.imshow(image.squeeze(), cmap='gray')
plt.title('Preprocessed Image (Label: {})'.format(label))
plt.axis('off')
# Show the plots
plt.show()
Images after Gaussian Blurring and Normalization
#Displaying images (randomly) from each category
num_images = 5
# Create a dictionary to store the indices of each category
category_indices = {}
for category in range(num_classes):
category_indices[category] = label_df[label_df['Label'] == category].index.tolist()
# Loop through each unique label
for label in name_classes:
# Get the indices of all images with the current label
indices = np.where(label_df['Label'] == label)[0]
# Choose num_images random indices
random_indices = np.random.choice(indices, size=num_images, replace=False)
# Get the corresponding images from "img_array"
images = img_array[random_indices]
# Create a plot of the images
fig, ax = plt.subplots(1, num_images, figsize=(20, 4))
for i, image in enumerate(images):
ax[i].imshow(image)
ax[i].axis('off')
# Set the suptitle and adjust the spacing
fig.suptitle(label, fontsize=16)
fig.subplots_adjust(top=1.1, bottom=0.1)
plt.show()
Output hidden; open in https://colab.research.google.com to view.
Beware that the differences in the appearance of the original image and the Gaussian-Normalized image are hard to notice by a human eye.
# Define a dictionary to map each label to a unique integer value
label_map = {'Small-flowered Cranesbill':0, 'Fat Hen':1, 'Shepherds Purse':2,
'Common wheat':3, 'Common Chickweed':4, 'Charlock':5, 'Cleavers':6,
'Scentless Mayweed':7, 'Sugar beet':8, 'Maize':9, 'Black-grass':10,
'Loose Silky-bent':11}
# Convert the label strings to integer values using the label map
label_df['Label'] = label_df['Label'].map(label_map)
#splitting data into training and testing set
X_train, X_test, y_train, y_test = train_test_split(img_array,label_df['Label'], test_size=0.1, random_state=42, stratify=label_df['Label'])
We can oversample the minority class by adding more samples to that minority classes doing augmentation, generating synthetic data, etc. We can undersample the majority class by removing samples from the majority class to balance the distribution. However we will lose information and although the distribution of the majority class to minitory class is 3 to 1, this may not be a good approach because our dataset is not very large. We can also use data augmentation. And we can also increase the weights of the minority classes.
The data is imbalanced as we have seen earlier. To avoid bias toward the majority class, or underfitting the minority class, we should balance the data.
Balancing the data on the Training set by adjusting weights.
The main reason to use this technique is because reprocessing the images have a very high computational cost.
# Calculate class weights to balance the data
class_weights = {}
for i in range(12):
class_weights[i] = np.sum(y_train == i) / len(y_train)
print(class_weights)
{0: 0.10432748538011696, 1: 0.09988304093567252, 2: 0.048654970760233916, 3: 0.04654970760233918, 4: 0.1286549707602339, 5: 0.08210526315789474, 6: 0.060350877192982454, 7: 0.10853801169590643, 8: 0.08116959064327485, 9: 0.04654970760233918, 10: 0.05543859649122807, 11: 0.13777777777777778}
# Use the class weights to balance the data
sample_weights = compute_sample_weight(class_weight=class_weights, y=y_train)
y_train.info
<bound method Series.info of 437 0
1571 4
838 1
4040 10
4707 11
..
746 1
1217 3
3058 7
4682 11
787 1
Name: Label, Length: 4275, dtype: int64>
# Convert the labels to one-hot vector (we only encode the y_train and y_test because these are categorical)
y_train_onehot = to_categorical(y_train, num_classes=12)
y_test_onehot = to_categorical(y_test, num_classes=12)
print("X_train shape:",X_train.shape)
print("X_test shape:",X_test.shape)
print("y_train shape:",y_train.shape)
print("y_test shape:",y_test.shape)
print("y_train one-hot encoded shape:",y_train_onehot.shape)
print("y_test one-hot encoded shape:",y_test_onehot.shape)
X_train shape: (4275, 128, 128, 3) X_test shape: (475, 128, 128, 3) y_train shape: (4275,) y_test shape: (475,) y_train one-hot encoded shape: (4275, 12) y_test one-hot encoded shape: (475, 12)
The data is compatible with Keras. img_array is a 4D array with shape (4750, 128, 128, 3), with 4750 images, sized 128x128px and the 3 RGB channels.
We already one hot encoded y_train and y_test, so these are also compatible with Keras.
input_shape = (128,128,3)
#create dataframe to store scores
Scoresdf = pd.DataFrame(columns =['model', 'accuracy', 'layers', 'total parameters', 'neurons', 'optimizer'])
We will first start with a simple model of only 1 Convolution layer.
Model Definition
Layers : 4
Layer 1: Convolutional layer : 32 filters, Kernel_size = (3,3), Relu
Layer 2: MaxPooling2D layer
Layer 3: Flatten layer
Layer 4: Dense layer : 12 neurons, Relu
Optimizer: Adam
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
# Define the model architecture
model0 = Sequential()
#Convolution with 32 filters size 3,3 and relu
model0.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=input_shape))
#add max pooling to reduce the size of output of first convolution layer
model0.add(MaxPooling2D(pool_size=(2, 2)))
#reshape the output to a flat vector
model0.add(Flatten())
#add output layer
model0.add(Dense(num_classes, activation='softmax'))
# Compile the model
model0.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
# Print the model summary
model0.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 126, 126, 32) 896
max_pooling2d (MaxPooling2D (None, 63, 63, 32) 0
)
flatten (Flatten) (None, 127008) 0
dense (Dense) (None, 12) 1524108
=================================================================
Total params: 1,525,004
Trainable params: 1,525,004
Non-trainable params: 0
_________________________________________________________________
Model Visualization
#visualize the CNN architecture
plot_model(model0, to_file='model0_plot.png', show_shapes=True, show_layer_names=True)
#visualize the CNN architecture using visualkeras
visualkeras.layered_view(model0, legend=True)
Early Stopping
Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 30 epochs and validation_split as 10%
history0 = model0.fit(X_train, y_train_onehot, epochs=30, batch_size=32,validation_split=0.10,callbacks=[es, mc])
Epoch 1/30 121/121 [==============================] - ETA: 0s - loss: 1.7578 - accuracy: 0.4195 Epoch 1: val_accuracy improved from -inf to 0.49299, saving model to best_model.h5 121/121 [==============================] - 11s 15ms/step - loss: 1.7578 - accuracy: 0.4195 - val_loss: 1.4395 - val_accuracy: 0.4930 Epoch 2/30 118/121 [============================>.] - ETA: 0s - loss: 1.1304 - accuracy: 0.6242 Epoch 2: val_accuracy improved from 0.49299 to 0.52570, saving model to best_model.h5 121/121 [==============================] - 1s 11ms/step - loss: 1.1293 - accuracy: 0.6252 - val_loss: 1.3579 - val_accuracy: 0.5257 Epoch 3/30 116/121 [===========================>..] - ETA: 0s - loss: 0.8899 - accuracy: 0.7077 Epoch 3: val_accuracy improved from 0.52570 to 0.56308, saving model to best_model.h5 121/121 [==============================] - 1s 12ms/step - loss: 0.8907 - accuracy: 0.7060 - val_loss: 1.3385 - val_accuracy: 0.5631 Epoch 4/30 118/121 [============================>.] - ETA: 0s - loss: 0.7389 - accuracy: 0.7630 Epoch 4: val_accuracy improved from 0.56308 to 0.61449, saving model to best_model.h5 121/121 [==============================] - 2s 13ms/step - loss: 0.7382 - accuracy: 0.7632 - val_loss: 1.1498 - val_accuracy: 0.6145 Epoch 5/30 120/121 [============================>.] - ETA: 0s - loss: 0.6034 - accuracy: 0.8198 Epoch 5: val_accuracy improved from 0.61449 to 0.63318, saving model to best_model.h5 121/121 [==============================] - 2s 20ms/step - loss: 0.6029 - accuracy: 0.8199 - val_loss: 1.1829 - val_accuracy: 0.6332 Epoch 6/30 121/121 [==============================] - ETA: 0s - loss: 0.4786 - accuracy: 0.8648 Epoch 6: val_accuracy improved from 0.63318 to 0.65187, saving model to best_model.h5 121/121 [==============================] - 3s 21ms/step - loss: 0.4786 - accuracy: 0.8648 - val_loss: 1.1076 - val_accuracy: 0.6519 Epoch 7/30 118/121 [============================>.] - ETA: 0s - loss: 0.3646 - accuracy: 0.9007 Epoch 7: val_accuracy did not improve from 0.65187 121/121 [==============================] - 2s 15ms/step - loss: 0.3628 - accuracy: 0.9012 - val_loss: 1.2155 - val_accuracy: 0.6145 Epoch 8/30 118/121 [============================>.] - ETA: 0s - loss: 0.2988 - accuracy: 0.9285 Epoch 8: val_accuracy did not improve from 0.65187 121/121 [==============================] - 1s 11ms/step - loss: 0.2991 - accuracy: 0.9283 - val_loss: 1.1235 - val_accuracy: 0.6379 Epoch 9/30 117/121 [============================>.] - ETA: 0s - loss: 0.2241 - accuracy: 0.9541 Epoch 9: val_accuracy did not improve from 0.65187 121/121 [==============================] - 1s 11ms/step - loss: 0.2233 - accuracy: 0.9548 - val_loss: 1.2661 - val_accuracy: 0.6285 Epoch 10/30 119/121 [============================>.] - ETA: 0s - loss: 0.1746 - accuracy: 0.9648 Epoch 10: val_accuracy did not improve from 0.65187 121/121 [==============================] - 1s 11ms/step - loss: 0.1754 - accuracy: 0.9644 - val_loss: 1.2412 - val_accuracy: 0.6145 Epoch 11/30 121/121 [==============================] - ETA: 0s - loss: 0.1371 - accuracy: 0.9789 Epoch 11: val_accuracy did not improve from 0.65187 121/121 [==============================] - 1s 11ms/step - loss: 0.1371 - accuracy: 0.9789 - val_loss: 1.2547 - val_accuracy: 0.6495 Epoch 12/30 118/121 [============================>.] - ETA: 0s - loss: 0.0958 - accuracy: 0.9897 Epoch 12: val_accuracy did not improve from 0.65187 121/121 [==============================] - 1s 10ms/step - loss: 0.0957 - accuracy: 0.9899 - val_loss: 1.3280 - val_accuracy: 0.6238 Epoch 13/30 117/121 [============================>.] - ETA: 0s - loss: 0.0735 - accuracy: 0.9925 Epoch 13: val_accuracy improved from 0.65187 to 0.65654, saving model to best_model.h5 121/121 [==============================] - 1s 11ms/step - loss: 0.0735 - accuracy: 0.9927 - val_loss: 1.2791 - val_accuracy: 0.6565 Epoch 14/30 119/121 [============================>.] - ETA: 0s - loss: 0.0515 - accuracy: 0.9968 Epoch 14: val_accuracy did not improve from 0.65654 121/121 [==============================] - 1s 10ms/step - loss: 0.0513 - accuracy: 0.9969 - val_loss: 1.3697 - val_accuracy: 0.6285 Epoch 15/30 117/121 [============================>.] - ETA: 0s - loss: 0.0368 - accuracy: 0.9989 Epoch 15: val_accuracy did not improve from 0.65654 121/121 [==============================] - 1s 12ms/step - loss: 0.0367 - accuracy: 0.9990 - val_loss: 1.3174 - val_accuracy: 0.6449 Epoch 16/30 118/121 [============================>.] - ETA: 0s - loss: 0.0315 - accuracy: 0.9987 Epoch 16: val_accuracy did not improve from 0.65654 121/121 [==============================] - 2s 13ms/step - loss: 0.0316 - accuracy: 0.9987 - val_loss: 1.3889 - val_accuracy: 0.6449 Epoch 16: early stopping
Notice that at Epoch 16 the training stopped since the validation loss had continued to increase during 10 epochs. The highest validation accuracy was 0.65654 and validation loss 1.2791.
Model0 - Accuracy vs Epoch curve
plt.plot(history0.history['accuracy'])
plt.plot(history0.history['val_accuracy'])
plt.title('model0 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
We can see that this model performs poorly. The model performs very well on training but very poorly in the validation. It is clearly overfitting.
Evaluation of model0 model.evaluate using the Test data
# Evaluate the model on the test data
score0 = model0.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score0[0])
print('Test accuracy:', score0[1])
Test loss: 1.5414196252822876 Test accuracy: 0.6547368168830872
# Test Prediction
y_pred = model0.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)
15/15 [==============================] - 0s 5ms/step
# Test Accuracy
accuracyScore0 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore0)
0.6547368421052632
Classification Report
# Compute the classification report
cr0 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)
print('Classification report:\n',cr0)
Classification report:
precision recall f1-score support
Small-flowered Cranesbill 0.79 0.82 0.80 50
Fat Hen 0.57 0.69 0.62 48
Shepherds Purse 0.40 0.17 0.24 23
Common wheat 0.41 0.32 0.36 22
Common Chickweed 0.75 0.79 0.77 61
Charlock 0.88 0.90 0.89 39
Cleavers 0.70 0.48 0.57 29
Scentless Mayweed 0.61 0.81 0.69 52
Sugar beet 0.71 0.45 0.55 38
Maize 0.58 0.50 0.54 22
Black-grass 0.36 0.46 0.41 26
Loose Silky-bent 0.68 0.72 0.70 65
accuracy 0.65 475
macro avg 0.62 0.59 0.60 475
weighted avg 0.65 0.65 0.65 475
Charlock has a higher precision, recall, and F1 score (0.88, 90, 0.89) than all other classes. Black-grass has the opposite (all low: 0.36, 0.46, 0.41).
Model0 has an overall accuracy of 0.65, it correctly predicted the seedling's class for 65% of the images in the test set.
The precision, recall, and F1-score for each class vary, with some classes having good performance (e.g., Charlock) and others having poor performance (e.g., Shepherds Purse).
The macro average F1 score is 0.65, which means that Model0 is not so good overall. The weighted average F1 score is also .65 and this means that the model is not biased towards any particular class.
Scores_Models_df = Scores_Models_df.append({'model':'Model0','accuracy':accuracyScore0,'layers':'4','total parameters':'1,525,004', 'neurons':'12', 'optimizer':'Adam'},ignore_index=True)
Confusion Matrix
# Set the confusion matrix
cm0 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
sns.heatmap(cm0, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()
Model0 is able to predict some classes with good accuracy but others are not predicted well. We can build better models to improve the performance on the other classes.
This model trained 1,525,004 parameters.
We called our first model, Model0 because this was a simple model with only one Convolution layer, so we didn't expect much from it.
Model Definition
This model builds on Model0. Model1 adds a second Convolutional layer and the fully connected layer has 128 neurons to improve the previous Model (Model0).
Layer 7: Droput layer: 0.5
Optimizer: Adam
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
Model Implementation
# Define the model architecture
model1 = Sequential()
#Convolution with 32 filters size 3,3 and relu
model1.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=input_shape))
#add max pooling to reduce the size of output of first convolution layer
model1.add(MaxPooling2D(pool_size=(2, 2)))
#Convolution with 64 filters size 3,3 and relu
model1.add(Conv2D(64, kernel_size=(3, 3), activation='relu'))
#add max pooling to reduce the size of output of second convolution layer
model1.add(MaxPooling2D(pool_size=(2, 2)))
#reshape the output to a flat vector
model1.add(Flatten())
#fully connected layer with 128 neurons
model1.add(Dense(128, activation='relu'))
#add dropout to prevent overfitting
model1.add(Dropout(0.5))
#add output layer
model1.add(Dense(num_classes, activation='softmax'))
# Compile the model
model1.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
# Print the model summary
model1.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 126, 126, 32) 896
max_pooling2d (MaxPooling2D (None, 63, 63, 32) 0
)
conv2d_1 (Conv2D) (None, 61, 61, 64) 18496
max_pooling2d_1 (MaxPooling (None, 30, 30, 64) 0
2D)
flatten (Flatten) (None, 57600) 0
dense (Dense) (None, 128) 7372928
dropout (Dropout) (None, 128) 0
dense_1 (Dense) (None, 12) 1548
=================================================================
Total params: 7,393,868
Trainable params: 7,393,868
Non-trainable params: 0
_________________________________________________________________
Model Visualization
#visualize the CNN architecture
plot_model(model1, to_file='model1_plot.png', show_shapes=True, show_layer_names=True)
#visualize the CNN architecture using visualkeras
visualkeras.layered_view(model1, legend=True)
Early Stopping
Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 60 epochs and validation_split as 10%
history1 = model1.fit(X_train,
y_train_onehot,
epochs=60,
batch_size=32,validation_split=0.10,callbacks=[es, mc])
Epoch 1/60 121/121 [==============================] - ETA: 0s - loss: 1.9516 - accuracy: 0.3465 Epoch 1: val_accuracy improved from -inf to 0.40654, saving model to best_model.h5 121/121 [==============================] - 6s 25ms/step - loss: 1.9516 - accuracy: 0.3465 - val_loss: 1.6409 - val_accuracy: 0.4065 Epoch 2/60 120/121 [============================>.] - ETA: 0s - loss: 1.4514 - accuracy: 0.5044 Epoch 2: val_accuracy improved from 0.40654 to 0.57477, saving model to best_model.h5 121/121 [==============================] - 2s 19ms/step - loss: 1.4530 - accuracy: 0.5048 - val_loss: 1.2422 - val_accuracy: 0.5748 Epoch 3/60 118/121 [============================>.] - ETA: 0s - loss: 1.2194 - accuracy: 0.5789 Epoch 3: val_accuracy improved from 0.57477 to 0.60748, saving model to best_model.h5 121/121 [==============================] - 2s 19ms/step - loss: 1.2182 - accuracy: 0.5797 - val_loss: 1.1769 - val_accuracy: 0.6075 Epoch 4/60 118/121 [============================>.] - ETA: 0s - loss: 1.0558 - accuracy: 0.6409 Epoch 4: val_accuracy improved from 0.60748 to 0.61682, saving model to best_model.h5 121/121 [==============================] - 2s 20ms/step - loss: 1.0574 - accuracy: 0.6397 - val_loss: 1.1029 - val_accuracy: 0.6168 Epoch 5/60 120/121 [============================>.] - ETA: 0s - loss: 0.9762 - accuracy: 0.6661 Epoch 5: val_accuracy improved from 0.61682 to 0.64720, saving model to best_model.h5 121/121 [==============================] - 3s 21ms/step - loss: 0.9757 - accuracy: 0.6662 - val_loss: 1.0191 - val_accuracy: 0.6472 Epoch 6/60 118/121 [============================>.] - ETA: 0s - loss: 0.8475 - accuracy: 0.7121 Epoch 6: val_accuracy improved from 0.64720 to 0.66121, saving model to best_model.h5 121/121 [==============================] - 2s 19ms/step - loss: 0.8487 - accuracy: 0.7115 - val_loss: 0.9492 - val_accuracy: 0.6612 Epoch 7/60 120/121 [============================>.] - ETA: 0s - loss: 0.7602 - accuracy: 0.7474 Epoch 7: val_accuracy did not improve from 0.66121 121/121 [==============================] - 2s 17ms/step - loss: 0.7595 - accuracy: 0.7476 - val_loss: 1.0222 - val_accuracy: 0.6519 Epoch 8/60 118/121 [============================>.] - ETA: 0s - loss: 0.6621 - accuracy: 0.7701 Epoch 8: val_accuracy improved from 0.66121 to 0.71729, saving model to best_model.h5 121/121 [==============================] - 2s 19ms/step - loss: 0.6596 - accuracy: 0.7705 - val_loss: 0.9412 - val_accuracy: 0.7173 Epoch 9/60 120/121 [============================>.] - ETA: 0s - loss: 0.5856 - accuracy: 0.7997 Epoch 9: val_accuracy improved from 0.71729 to 0.73131, saving model to best_model.h5 121/121 [==============================] - 2s 18ms/step - loss: 0.5852 - accuracy: 0.8001 - val_loss: 0.9632 - val_accuracy: 0.7313 Epoch 10/60 118/121 [============================>.] - ETA: 0s - loss: 0.5189 - accuracy: 0.8234 Epoch 10: val_accuracy did not improve from 0.73131 121/121 [==============================] - 2s 18ms/step - loss: 0.5186 - accuracy: 0.8230 - val_loss: 0.9905 - val_accuracy: 0.7173 Epoch 11/60 120/121 [============================>.] - ETA: 0s - loss: 0.4564 - accuracy: 0.8458 Epoch 11: val_accuracy did not improve from 0.73131 121/121 [==============================] - 2s 18ms/step - loss: 0.4575 - accuracy: 0.8456 - val_loss: 0.9526 - val_accuracy: 0.7150 Epoch 12/60 121/121 [==============================] - ETA: 0s - loss: 0.4149 - accuracy: 0.8513 Epoch 12: val_accuracy improved from 0.73131 to 0.73364, saving model to best_model.h5 121/121 [==============================] - 2s 19ms/step - loss: 0.4149 - accuracy: 0.8513 - val_loss: 0.9764 - val_accuracy: 0.7336 Epoch 13/60 119/121 [============================>.] - ETA: 0s - loss: 0.3751 - accuracy: 0.8661 Epoch 13: val_accuracy did not improve from 0.73364 121/121 [==============================] - 2s 16ms/step - loss: 0.3761 - accuracy: 0.8659 - val_loss: 0.9244 - val_accuracy: 0.7313 Epoch 14/60 121/121 [==============================] - ETA: 0s - loss: 0.3564 - accuracy: 0.8768 Epoch 14: val_accuracy did not improve from 0.73364 121/121 [==============================] - 2s 17ms/step - loss: 0.3564 - accuracy: 0.8768 - val_loss: 0.9590 - val_accuracy: 0.7243 Epoch 15/60 120/121 [============================>.] - ETA: 0s - loss: 0.3181 - accuracy: 0.8914 Epoch 15: val_accuracy did not improve from 0.73364 121/121 [==============================] - 2s 17ms/step - loss: 0.3178 - accuracy: 0.8916 - val_loss: 0.9571 - val_accuracy: 0.7290 Epoch 16/60 121/121 [==============================] - ETA: 0s - loss: 0.2852 - accuracy: 0.9041 Epoch 16: val_accuracy did not improve from 0.73364 121/121 [==============================] - 2s 18ms/step - loss: 0.2852 - accuracy: 0.9041 - val_loss: 1.1569 - val_accuracy: 0.7079 Epoch 17/60 120/121 [============================>.] - ETA: 0s - loss: 0.2488 - accuracy: 0.9107 Epoch 17: val_accuracy did not improve from 0.73364 121/121 [==============================] - 2s 19ms/step - loss: 0.2490 - accuracy: 0.9106 - val_loss: 1.1628 - val_accuracy: 0.7266 Epoch 18/60 118/121 [============================>.] - ETA: 0s - loss: 0.2405 - accuracy: 0.9150 Epoch 18: val_accuracy did not improve from 0.73364 121/121 [==============================] - 2s 18ms/step - loss: 0.2401 - accuracy: 0.9147 - val_loss: 1.1033 - val_accuracy: 0.7243 Epoch 19/60 118/121 [============================>.] - ETA: 0s - loss: 0.1999 - accuracy: 0.9311 Epoch 19: val_accuracy did not improve from 0.73364 121/121 [==============================] - 2s 17ms/step - loss: 0.1992 - accuracy: 0.9314 - val_loss: 1.3685 - val_accuracy: 0.7079 Epoch 20/60 118/121 [============================>.] - ETA: 0s - loss: 0.1953 - accuracy: 0.9288 Epoch 20: val_accuracy did not improve from 0.73364 121/121 [==============================] - 2s 17ms/step - loss: 0.1968 - accuracy: 0.9280 - val_loss: 1.1961 - val_accuracy: 0.7336 Epoch 21/60 120/121 [============================>.] - ETA: 0s - loss: 0.1952 - accuracy: 0.9312 Epoch 21: val_accuracy did not improve from 0.73364 121/121 [==============================] - 2s 17ms/step - loss: 0.1949 - accuracy: 0.9314 - val_loss: 1.2831 - val_accuracy: 0.7196 Epoch 22/60 118/121 [============================>.] - ETA: 0s - loss: 0.1820 - accuracy: 0.9370 Epoch 22: val_accuracy improved from 0.73364 to 0.75000, saving model to best_model.h5 121/121 [==============================] - 2s 18ms/step - loss: 0.1821 - accuracy: 0.9371 - val_loss: 1.1866 - val_accuracy: 0.7500 Epoch 23/60 119/121 [============================>.] - ETA: 0s - loss: 0.1904 - accuracy: 0.9338 Epoch 23: val_accuracy did not improve from 0.75000 121/121 [==============================] - 2s 19ms/step - loss: 0.1899 - accuracy: 0.9340 - val_loss: 1.2181 - val_accuracy: 0.7290 Epoch 23: early stopping
Notice that at Epoch 23 the training stopped since the validation loss had continued to increase during 10 epochs. The highest validation accuracy was 0.75 and validation loss of 1.1866.
Model1 - Accuracy vs Epoch curve
plt.plot(history1.history['accuracy'])
plt.plot(history1.history['val_accuracy'])
plt.title('model1 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
Notice that the model shows overfitting. Past epoch 12 the validation loss doesn't reduce or begins to increase.
The model performs poorly as we see that the training accuracy is high but the validation accuracy is poor, indicating that the model is overfitting.
Evaluation of model1 model.evaluate using the Test data
# Evaluate the model on the test data
score1 = model1.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score1[0])
print('Test accuracy:', score1[1])
Test loss: 1.2476131916046143 Test accuracy: 0.7326315641403198
# Test Prediction
y_pred = model1.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)
15/15 [==============================] - 0s 8ms/step
# Test Accuracy
accuracyScore1 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore1)
0.7326315789473684
Classification Report
# Compute the classification report
cr1 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)
print('Classification report:\n',cr1)
Classification report:
precision recall f1-score support
Small-flowered Cranesbill 0.85 0.92 0.88 50
Fat Hen 0.81 0.73 0.77 48
Shepherds Purse 0.64 0.30 0.41 23
Common wheat 0.71 0.55 0.62 22
Common Chickweed 0.83 0.89 0.86 61
Charlock 0.69 0.97 0.81 39
Cleavers 0.74 0.69 0.71 29
Scentless Mayweed 0.71 0.67 0.69 52
Sugar beet 0.84 0.68 0.75 38
Maize 0.65 0.68 0.67 22
Black-grass 0.36 0.38 0.37 26
Loose Silky-bent 0.69 0.77 0.73 65
accuracy 0.73 475
macro avg 0.71 0.69 0.69 475
weighted avg 0.73 0.73 0.73 475
The model seems to perform well for classes such as Small-flowered Cranesbill, Common Chickweed, and Charlock with high precision, recall, and F1-score.
The model performs poorly with classes like Black-grass, Shepherds Purse, and Common wheat that have low precision, recall, and F1-score.
The macro-average and weighted-average F1-score are 0.69 and 0.73, which indicates that the model has an overall decent performance.
However, this model can be improved, especially for the clasess that perform poorly.
Scoresdf = Scoresdf.append({'model':'Model1','accuracy':accuracyScore1,'layers':'8','total parameters':'7,393,868', 'neurons':'128', 'optimizer':'Adam'},ignore_index=True)
Confusion Matrix
# Set the confusion matrix
cm1 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
sns.heatmap(cm1, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()
Model1 one performs better than Model0 but the model isn't very good either. It trains 7,393,868 parameters which is about 4 times more parameters than Model0 and although this difference is massive, Model1 doesn't have a great performance either.
Model2 keeps growing, to try to improve on the previous model (Model1) we added more layers.
Layers : 10
Layer 1: Convolutional layer : 32 filters, Kernel_size = (3,3), Relu
Layer 9: Droput layer: 0.5
Optimizer: Adam
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
Model Implementation
# Define the model architecture
model2 = Sequential()
#Convolution with 32 filters size 3,3 and relu
model2.add(Conv2D(32, kernel_size=(3,3), activation='relu', input_shape=(128,128,3)))
#add max pooling to reduce the size of output of first convolution layer
model2.add(MaxPooling2D(pool_size=(2,2)))
#Convolution with 64 filters size 3,3 and relu
model2.add(Conv2D(64, kernel_size=(3,3), activation='relu'))
#add max pooling to reduce the size of output of first convolution layer
model2.add(MaxPooling2D(pool_size=(2,2)))
#Convolution with 128 filters size 3,3 and relu
model2.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
#add max pooling to reduce the size of output of first convolution layer
model2.add(MaxPooling2D(pool_size=(2,2)))
#reshape the output to a flat vector
model2.add(Flatten())
#fully connected layer with 128 neurons
model2.add(Dense(128, activation='relu'))
#add dropout to prevent overfitting
model2.add(Dropout(0.5))
#add output layer
model2.add(Dense(num_classes, activation='softmax'))
# Compile the model
model2.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
# Print the model summary
model2.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 126, 126, 32) 896
max_pooling2d (MaxPooling2D (None, 63, 63, 32) 0
)
conv2d_1 (Conv2D) (None, 61, 61, 64) 18496
max_pooling2d_1 (MaxPooling (None, 30, 30, 64) 0
2D)
conv2d_2 (Conv2D) (None, 28, 28, 128) 73856
max_pooling2d_2 (MaxPooling (None, 14, 14, 128) 0
2D)
flatten (Flatten) (None, 25088) 0
dense (Dense) (None, 128) 3211392
dropout (Dropout) (None, 128) 0
dense_1 (Dense) (None, 12) 1548
=================================================================
Total params: 3,306,188
Trainable params: 3,306,188
Non-trainable params: 0
_________________________________________________________________
Model Visualization
#visualize the cnn model architecture
plot_model(model2, to_file='model2_plot.png', show_shapes=True, show_layer_names=True)
#visualize the cnn model architecture using visualkeras
visualkeras.layered_view(model2, legend=True)
Early Stopping
Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 40 epochs and validation_split as 10%
history2 = model2.fit(X_train,
y_train_onehot,
epochs=40,
batch_size=32,validation_split=0.10,callbacks=[es, mc])
Epoch 1/40 121/121 [==============================] - ETA: 0s - loss: 1.9661 - accuracy: 0.3408 Epoch 1: val_accuracy improved from -inf to 0.46729, saving model to best_model.h5 121/121 [==============================] - 5s 23ms/step - loss: 1.9661 - accuracy: 0.3408 - val_loss: 1.5358 - val_accuracy: 0.4673 Epoch 2/40 118/121 [============================>.] - ETA: 0s - loss: 1.4708 - accuracy: 0.4955 Epoch 2: val_accuracy improved from 0.46729 to 0.58178, saving model to best_model.h5 121/121 [==============================] - 2s 20ms/step - loss: 1.4663 - accuracy: 0.4975 - val_loss: 1.2714 - val_accuracy: 0.5818 Epoch 3/40 118/121 [============================>.] - ETA: 0s - loss: 1.2480 - accuracy: 0.5731 Epoch 3: val_accuracy improved from 0.58178 to 0.60280, saving model to best_model.h5 121/121 [==============================] - 3s 22ms/step - loss: 1.2457 - accuracy: 0.5745 - val_loss: 1.1380 - val_accuracy: 0.6028 Epoch 4/40 118/121 [============================>.] - ETA: 0s - loss: 1.0826 - accuracy: 0.6253 Epoch 4: val_accuracy did not improve from 0.60280 121/121 [==============================] - 2s 19ms/step - loss: 1.0815 - accuracy: 0.6252 - val_loss: 1.0708 - val_accuracy: 0.6028 Epoch 5/40 118/121 [============================>.] - ETA: 0s - loss: 1.0035 - accuracy: 0.6528 Epoch 5: val_accuracy improved from 0.60280 to 0.71262, saving model to best_model.h5 121/121 [==============================] - 2s 20ms/step - loss: 1.0038 - accuracy: 0.6525 - val_loss: 0.8658 - val_accuracy: 0.7126 Epoch 6/40 119/121 [============================>.] - ETA: 0s - loss: 0.9040 - accuracy: 0.6875 Epoch 6: val_accuracy improved from 0.71262 to 0.73832, saving model to best_model.h5 121/121 [==============================] - 2s 20ms/step - loss: 0.9051 - accuracy: 0.6875 - val_loss: 0.8284 - val_accuracy: 0.7383 Epoch 7/40 120/121 [============================>.] - ETA: 0s - loss: 0.8123 - accuracy: 0.7240 Epoch 7: val_accuracy did not improve from 0.73832 121/121 [==============================] - 2s 18ms/step - loss: 0.8115 - accuracy: 0.7245 - val_loss: 0.8342 - val_accuracy: 0.7336 Epoch 8/40 119/121 [============================>.] - ETA: 0s - loss: 0.7744 - accuracy: 0.7269 Epoch 8: val_accuracy improved from 0.73832 to 0.75000, saving model to best_model.h5 121/121 [==============================] - 2s 21ms/step - loss: 0.7721 - accuracy: 0.7271 - val_loss: 0.7839 - val_accuracy: 0.7500 Epoch 9/40 118/121 [============================>.] - ETA: 0s - loss: 0.6741 - accuracy: 0.7579 Epoch 9: val_accuracy improved from 0.75000 to 0.75935, saving model to best_model.h5 121/121 [==============================] - 3s 21ms/step - loss: 0.6752 - accuracy: 0.7575 - val_loss: 0.7541 - val_accuracy: 0.7593 Epoch 10/40 119/121 [============================>.] - ETA: 0s - loss: 0.6485 - accuracy: 0.7731 Epoch 10: val_accuracy did not improve from 0.75935 121/121 [==============================] - 2s 18ms/step - loss: 0.6509 - accuracy: 0.7710 - val_loss: 0.8598 - val_accuracy: 0.7150 Epoch 11/40 118/121 [============================>.] - ETA: 0s - loss: 0.6068 - accuracy: 0.7858 Epoch 11: val_accuracy improved from 0.75935 to 0.77336, saving model to best_model.h5 121/121 [==============================] - 2s 19ms/step - loss: 0.6083 - accuracy: 0.7845 - val_loss: 0.7194 - val_accuracy: 0.7734 Epoch 12/40 118/121 [============================>.] - ETA: 0s - loss: 0.5451 - accuracy: 0.8014 Epoch 12: val_accuracy did not improve from 0.77336 121/121 [==============================] - 2s 18ms/step - loss: 0.5444 - accuracy: 0.8019 - val_loss: 0.7628 - val_accuracy: 0.7617 Epoch 13/40 120/121 [============================>.] - ETA: 0s - loss: 0.5140 - accuracy: 0.8120 Epoch 13: val_accuracy improved from 0.77336 to 0.77570, saving model to best_model.h5 121/121 [==============================] - 2s 20ms/step - loss: 0.5143 - accuracy: 0.8118 - val_loss: 0.7286 - val_accuracy: 0.7757 Epoch 14/40 118/121 [============================>.] - ETA: 0s - loss: 0.4719 - accuracy: 0.8247 Epoch 14: val_accuracy improved from 0.77570 to 0.78505, saving model to best_model.h5 121/121 [==============================] - 3s 21ms/step - loss: 0.4699 - accuracy: 0.8256 - val_loss: 0.7710 - val_accuracy: 0.7850 Epoch 15/40 118/121 [============================>.] - ETA: 0s - loss: 0.4808 - accuracy: 0.8249 Epoch 15: val_accuracy did not improve from 0.78505 121/121 [==============================] - 2s 20ms/step - loss: 0.4792 - accuracy: 0.8253 - val_loss: 0.7628 - val_accuracy: 0.7850 Epoch 16/40 119/121 [============================>.] - ETA: 0s - loss: 0.4376 - accuracy: 0.8309 Epoch 16: val_accuracy improved from 0.78505 to 0.80841, saving model to best_model.h5 121/121 [==============================] - 2s 19ms/step - loss: 0.4375 - accuracy: 0.8310 - val_loss: 0.7322 - val_accuracy: 0.8084 Epoch 17/40 119/121 [============================>.] - ETA: 0s - loss: 0.4054 - accuracy: 0.8430 Epoch 17: val_accuracy did not improve from 0.80841 121/121 [==============================] - 2s 19ms/step - loss: 0.4051 - accuracy: 0.8427 - val_loss: 0.7825 - val_accuracy: 0.8061 Epoch 18/40 120/121 [============================>.] - ETA: 0s - loss: 0.4256 - accuracy: 0.8430 Epoch 18: val_accuracy did not improve from 0.80841 121/121 [==============================] - 2s 18ms/step - loss: 0.4252 - accuracy: 0.8430 - val_loss: 0.7699 - val_accuracy: 0.7897 Epoch 19/40 118/121 [============================>.] - ETA: 0s - loss: 0.3703 - accuracy: 0.8538 Epoch 19: val_accuracy did not improve from 0.80841 121/121 [==============================] - 2s 19ms/step - loss: 0.3703 - accuracy: 0.8539 - val_loss: 0.8140 - val_accuracy: 0.7757 Epoch 20/40 121/121 [==============================] - ETA: 0s - loss: 0.3800 - accuracy: 0.8550 Epoch 20: val_accuracy did not improve from 0.80841 121/121 [==============================] - 2s 20ms/step - loss: 0.3800 - accuracy: 0.8550 - val_loss: 0.8642 - val_accuracy: 0.7664 Epoch 21/40 118/121 [============================>.] - ETA: 0s - loss: 0.3413 - accuracy: 0.8636 Epoch 21: val_accuracy did not improve from 0.80841 121/121 [==============================] - 2s 19ms/step - loss: 0.3401 - accuracy: 0.8640 - val_loss: 0.8554 - val_accuracy: 0.7967 Epoch 21: early stopping
The training stopped at epoch 21 because the validation loss had increased during 10 epochs. The highest validation accuracy was 0.780 and validation loss 0.7628.
Model 2: Accuracy vs epochs plot
plt.plot(history2.history['accuracy'])
plt.plot(history2.history['val_accuracy'])
plt.title('model2 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
Notice that somewhere around epoch 15 the validation loss begins to increase.
Overall Model2 seems to perform better than the 2 previous models.
Evaluation of model2 model.evaluate using the Test data
# Evaluate the model on the test data
score2 = model2.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score2[0])
print('Test accuracy:', score2[1])
Test loss: 0.9052645564079285 Test accuracy: 0.7915789484977722
# Test Prediction
y_pred = model2.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)
15/15 [==============================] - 0s 7ms/step
# Test Accuracy
accuracyScore2 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore2)
0.791578947368421
Classification Report
# Compute the classification report
cr2 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)
print('Classification report:\n',cr2)
Classification report:
precision recall f1-score support
Small-flowered Cranesbill 0.88 0.88 0.88 50
Fat Hen 0.86 0.88 0.87 48
Shepherds Purse 0.85 0.48 0.61 23
Common wheat 0.61 0.86 0.72 22
Common Chickweed 0.83 0.95 0.89 61
Charlock 0.88 0.92 0.90 39
Cleavers 0.83 0.83 0.83 29
Scentless Mayweed 0.68 0.81 0.74 52
Sugar beet 0.87 0.71 0.78 38
Maize 0.61 0.64 0.62 22
Black-grass 0.52 0.46 0.49 26
Loose Silky-bent 0.89 0.72 0.80 65
accuracy 0.79 475
macro avg 0.77 0.76 0.76 475
weighted avg 0.80 0.79 0.79 475
The overall accuracy of Model2 is 0.79, which is a ok. The precision and recall values for each class are also mostly good, with some classes having higher values than others.
The highest precision and recall values are achieved for "Small-flowered Cranesbill", "Fat Hen", "Common Chickweed", "Charlock", and "Cleavers", which means that the model can identify well these classes.
Some classes have lower precision and recall values, such as "Shepherds Purse", "Common wheat", "Scentless Mayweed", "Sugar beet", "Maize", and "Black-grass".
Scoresdf = Scoresdf.append({'model':'Model2','accuracy':accuracyScore2,'layers':'10','total parameters':'3,306,188', 'neurons':'128', 'optimizer':'Adam'},ignore_index=True)
Confusion Matrix
#set and plot the Confusion matrix
cm2 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
sns.heatmap(cm2, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()
Model2 performs better than Model0 and Model1. It actually performs OK.
But we can improve the model for better performance.
To try to improve Model2, we change the optimizer from Adam to SGD.
Layers : 10
Layer 1: Convolutional layer : 32 filters, Kernel_size = (3,3), Relu
Layer 9: Droput layer: 0.5
Optimizer: SGD
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
Lets used SGD as optimizer with the second model since we obtained a higher accuracy than the first model
# Using SGD Optimizer
sgd = SGD(learning_rate=0.01, momentum=0.9)
Model Implementation
# Define the model architecture
model3 = Sequential()
#Convolution with 32 filters size 3,3 and relu
model3.add(Conv2D(32, kernel_size=(3,3), activation='relu', input_shape=(128,128,3)))
#add max pooling to reduce the size of output of first convolution layer
model3.add(MaxPooling2D(pool_size=(2,2)))
#Convolution with 64 filters size 3,3 and relu
model3.add(Conv2D(64, kernel_size=(3,3), activation='relu'))
#add max pooling to reduce the size of output of first convolution layer
model3.add(MaxPooling2D(pool_size=(2,2)))
#Convolution with 128 filters size 3,3 and relu
model3.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
#add max pooling to reduce the size of output of first convolution layer
model3.add(MaxPooling2D(pool_size=(2,2)))
#reshape the output to a flat vector
model3.add(Flatten())
#fully connected layer with 128 neurons
model3.add(Dense(128, activation='relu'))
#add dropout to prevent overfitting
model3.add(Dropout(0.5))
#add output layer
model3.add(Dense(num_classes, activation='softmax'))
# Compile the model
model3.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])
# Print the model summary
model3.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 126, 126, 32) 896
max_pooling2d (MaxPooling2D (None, 63, 63, 32) 0
)
conv2d_1 (Conv2D) (None, 61, 61, 64) 18496
max_pooling2d_1 (MaxPooling (None, 30, 30, 64) 0
2D)
conv2d_2 (Conv2D) (None, 28, 28, 128) 73856
max_pooling2d_2 (MaxPooling (None, 14, 14, 128) 0
2D)
flatten (Flatten) (None, 25088) 0
dense (Dense) (None, 128) 3211392
dropout (Dropout) (None, 128) 0
dense_1 (Dense) (None, 12) 1548
=================================================================
Total params: 3,306,188
Trainable params: 3,306,188
Non-trainable params: 0
_________________________________________________________________
Model Visualization
#visualize the cnn model architecture
plot_model(model3, to_file='model3_plot.png', show_shapes=True, show_layer_names=True)
#visualize the cnn model architecture using visualkeras
visualkeras.layered_view(model3, legend=True)
Early Stopping
Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 40 epochs and validation_split as 10%
history3 = model3.fit(X_train, y_train_onehot, epochs=40, batch_size=32,validation_split=0.10,callbacks=[es, mc])
Epoch 1/40 120/121 [============================>.] - ETA: 0s - loss: 2.3845 - accuracy: 0.1630 Epoch 1: val_accuracy improved from -inf to 0.33178, saving model to best_model.h5 121/121 [==============================] - 3s 24ms/step - loss: 2.3832 - accuracy: 0.1638 - val_loss: 2.0441 - val_accuracy: 0.3318 Epoch 2/40 118/121 [============================>.] - ETA: 0s - loss: 1.8634 - accuracy: 0.3557 Epoch 2: val_accuracy improved from 0.33178 to 0.36916, saving model to best_model.h5 121/121 [==============================] - 3s 22ms/step - loss: 1.8603 - accuracy: 0.3569 - val_loss: 1.7121 - val_accuracy: 0.3692 Epoch 3/40 118/121 [============================>.] - ETA: 0s - loss: 1.6466 - accuracy: 0.4198 Epoch 3: val_accuracy improved from 0.36916 to 0.47897, saving model to best_model.h5 121/121 [==============================] - 2s 19ms/step - loss: 1.6427 - accuracy: 0.4211 - val_loss: 1.5588 - val_accuracy: 0.4790 Epoch 4/40 118/121 [============================>.] - ETA: 0s - loss: 1.4886 - accuracy: 0.4738 Epoch 4: val_accuracy did not improve from 0.47897 121/121 [==============================] - 2s 19ms/step - loss: 1.4835 - accuracy: 0.4760 - val_loss: 1.5327 - val_accuracy: 0.4533 Epoch 5/40 118/121 [============================>.] - ETA: 0s - loss: 1.3412 - accuracy: 0.5368 Epoch 5: val_accuracy improved from 0.47897 to 0.53037, saving model to best_model.h5 121/121 [==============================] - 2s 19ms/step - loss: 1.3435 - accuracy: 0.5370 - val_loss: 1.4149 - val_accuracy: 0.5304 Epoch 6/40 118/121 [============================>.] - ETA: 0s - loss: 1.2239 - accuracy: 0.5646 Epoch 6: val_accuracy improved from 0.53037 to 0.56542, saving model to best_model.h5 121/121 [==============================] - 2s 19ms/step - loss: 1.2235 - accuracy: 0.5659 - val_loss: 1.2519 - val_accuracy: 0.5654 Epoch 7/40 118/121 [============================>.] - ETA: 0s - loss: 1.1519 - accuracy: 0.6004 Epoch 7: val_accuracy improved from 0.56542 to 0.60514, saving model to best_model.h5 121/121 [==============================] - 2s 20ms/step - loss: 1.1504 - accuracy: 0.6007 - val_loss: 1.0924 - val_accuracy: 0.6051 Epoch 8/40 118/121 [============================>.] - ETA: 0s - loss: 1.0355 - accuracy: 0.6412 Epoch 8: val_accuracy improved from 0.60514 to 0.66355, saving model to best_model.h5 121/121 [==============================] - 3s 21ms/step - loss: 1.0330 - accuracy: 0.6415 - val_loss: 0.9952 - val_accuracy: 0.6636 Epoch 9/40 120/121 [============================>.] - ETA: 0s - loss: 0.9334 - accuracy: 0.6693 Epoch 9: val_accuracy improved from 0.66355 to 0.69393, saving model to best_model.h5 121/121 [==============================] - 2s 20ms/step - loss: 0.9325 - accuracy: 0.6696 - val_loss: 0.9063 - val_accuracy: 0.6939 Epoch 10/40 120/121 [============================>.] - ETA: 0s - loss: 0.8660 - accuracy: 0.7000 Epoch 10: val_accuracy improved from 0.69393 to 0.71963, saving model to best_model.h5 121/121 [==============================] - 2s 19ms/step - loss: 0.8655 - accuracy: 0.7003 - val_loss: 0.8607 - val_accuracy: 0.7196 Epoch 11/40 121/121 [==============================] - ETA: 0s - loss: 0.7987 - accuracy: 0.7091 Epoch 11: val_accuracy did not improve from 0.71963 121/121 [==============================] - 2s 18ms/step - loss: 0.7987 - accuracy: 0.7091 - val_loss: 0.9272 - val_accuracy: 0.7009 Epoch 12/40 119/121 [============================>.] - ETA: 0s - loss: 0.7245 - accuracy: 0.7455 Epoch 12: val_accuracy did not improve from 0.71963 121/121 [==============================] - 2s 18ms/step - loss: 0.7236 - accuracy: 0.7463 - val_loss: 1.0322 - val_accuracy: 0.6846 Epoch 13/40 120/121 [============================>.] - ETA: 0s - loss: 0.6609 - accuracy: 0.7698 Epoch 13: val_accuracy improved from 0.71963 to 0.73131, saving model to best_model.h5 121/121 [==============================] - 3s 21ms/step - loss: 0.6606 - accuracy: 0.7697 - val_loss: 0.8166 - val_accuracy: 0.7313 Epoch 14/40 118/121 [============================>.] - ETA: 0s - loss: 0.6478 - accuracy: 0.7699 Epoch 14: val_accuracy did not improve from 0.73131 121/121 [==============================] - 2s 19ms/step - loss: 0.6486 - accuracy: 0.7705 - val_loss: 0.9199 - val_accuracy: 0.7103 Epoch 15/40 121/121 [==============================] - ETA: 0s - loss: 0.6651 - accuracy: 0.7583 Epoch 15: val_accuracy improved from 0.73131 to 0.73598, saving model to best_model.h5 121/121 [==============================] - 2s 19ms/step - loss: 0.6651 - accuracy: 0.7583 - val_loss: 0.7705 - val_accuracy: 0.7360 Epoch 16/40 118/121 [============================>.] - ETA: 0s - loss: 0.5159 - accuracy: 0.8136 Epoch 16: val_accuracy improved from 0.73598 to 0.77804, saving model to best_model.h5 121/121 [==============================] - 2s 19ms/step - loss: 0.5160 - accuracy: 0.8126 - val_loss: 0.7808 - val_accuracy: 0.7780 Epoch 17/40 118/121 [============================>.] - ETA: 0s - loss: 0.5104 - accuracy: 0.8178 Epoch 17: val_accuracy did not improve from 0.77804 121/121 [==============================] - 2s 19ms/step - loss: 0.5085 - accuracy: 0.8178 - val_loss: 0.9762 - val_accuracy: 0.7407 Epoch 18/40 120/121 [============================>.] - ETA: 0s - loss: 0.4982 - accuracy: 0.8117 Epoch 18: val_accuracy did not improve from 0.77804 121/121 [==============================] - 2s 19ms/step - loss: 0.4982 - accuracy: 0.8118 - val_loss: 0.8138 - val_accuracy: 0.7407 Epoch 19/40 118/121 [============================>.] - ETA: 0s - loss: 0.4704 - accuracy: 0.8271 Epoch 19: val_accuracy did not improve from 0.77804 121/121 [==============================] - 2s 21ms/step - loss: 0.4703 - accuracy: 0.8271 - val_loss: 0.8228 - val_accuracy: 0.7523 Epoch 20/40 120/121 [============================>.] - ETA: 0s - loss: 0.4214 - accuracy: 0.8461 Epoch 20: val_accuracy did not improve from 0.77804 121/121 [==============================] - 2s 19ms/step - loss: 0.4213 - accuracy: 0.8461 - val_loss: 0.8157 - val_accuracy: 0.7570 Epoch 21/40 119/121 [============================>.] - ETA: 0s - loss: 0.3918 - accuracy: 0.8522 Epoch 21: val_accuracy improved from 0.77804 to 0.78271, saving model to best_model.h5 121/121 [==============================] - 2s 19ms/step - loss: 0.3926 - accuracy: 0.8526 - val_loss: 0.7604 - val_accuracy: 0.7827 Epoch 22/40 118/121 [============================>.] - ETA: 0s - loss: 0.3737 - accuracy: 0.8623 Epoch 22: val_accuracy improved from 0.78271 to 0.79673, saving model to best_model.h5 121/121 [==============================] - 2s 19ms/step - loss: 0.3730 - accuracy: 0.8628 - val_loss: 0.7455 - val_accuracy: 0.7967 Epoch 23/40 120/121 [============================>.] - ETA: 0s - loss: 0.3372 - accuracy: 0.8784 Epoch 23: val_accuracy did not improve from 0.79673 121/121 [==============================] - 2s 18ms/step - loss: 0.3377 - accuracy: 0.8783 - val_loss: 0.8077 - val_accuracy: 0.7897 Epoch 24/40 119/121 [============================>.] - ETA: 0s - loss: 0.2710 - accuracy: 0.8981 Epoch 24: val_accuracy did not improve from 0.79673 121/121 [==============================] - 2s 19ms/step - loss: 0.2701 - accuracy: 0.8984 - val_loss: 0.9609 - val_accuracy: 0.7383 Epoch 25/40 120/121 [============================>.] - ETA: 0s - loss: 0.2583 - accuracy: 0.9003 Epoch 25: val_accuracy did not improve from 0.79673 121/121 [==============================] - 3s 25ms/step - loss: 0.2591 - accuracy: 0.9002 - val_loss: 0.8895 - val_accuracy: 0.7664 Epoch 26/40 119/121 [============================>.] - ETA: 0s - loss: 0.2844 - accuracy: 0.8968 Epoch 26: val_accuracy did not improve from 0.79673 121/121 [==============================] - 2s 20ms/step - loss: 0.2854 - accuracy: 0.8965 - val_loss: 0.9039 - val_accuracy: 0.7710 Epoch 27/40 119/121 [============================>.] - ETA: 0s - loss: 0.2522 - accuracy: 0.9099 Epoch 27: val_accuracy did not improve from 0.79673 121/121 [==============================] - 2s 19ms/step - loss: 0.2514 - accuracy: 0.9103 - val_loss: 0.8382 - val_accuracy: 0.7850 Epoch 28/40 120/121 [============================>.] - ETA: 0s - loss: 0.2276 - accuracy: 0.9177 Epoch 28: val_accuracy did not improve from 0.79673 121/121 [==============================] - 2s 18ms/step - loss: 0.2272 - accuracy: 0.9179 - val_loss: 1.0102 - val_accuracy: 0.7453 Epoch 29/40 118/121 [============================>.] - ETA: 0s - loss: 0.2346 - accuracy: 0.9155 Epoch 29: val_accuracy did not improve from 0.79673 121/121 [==============================] - 2s 19ms/step - loss: 0.2348 - accuracy: 0.9158 - val_loss: 0.7799 - val_accuracy: 0.7944 Epoch 30/40 118/121 [============================>.] - ETA: 0s - loss: 0.2243 - accuracy: 0.9150 Epoch 30: val_accuracy did not improve from 0.79673 121/121 [==============================] - 2s 20ms/step - loss: 0.2231 - accuracy: 0.9155 - val_loss: 0.8691 - val_accuracy: 0.7921 Epoch 31/40 121/121 [==============================] - ETA: 0s - loss: 0.2006 - accuracy: 0.9285 Epoch 31: val_accuracy did not improve from 0.79673 121/121 [==============================] - 2s 20ms/step - loss: 0.2006 - accuracy: 0.9285 - val_loss: 0.9503 - val_accuracy: 0.7687 Epoch 32/40 119/121 [============================>.] - ETA: 0s - loss: 0.1553 - accuracy: 0.9446 Epoch 32: val_accuracy improved from 0.79673 to 0.81542, saving model to best_model.h5 121/121 [==============================] - 2s 19ms/step - loss: 0.1540 - accuracy: 0.9452 - val_loss: 0.7941 - val_accuracy: 0.8154 Epoch 32: early stopping
Training stopped on epoch 28. The highest validation accuracy was 0.7921 where the validation loss was 0.844.
Model3 Accuracy vs Epoch curve
plt.plot(history3.history['accuracy'])
plt.plot(history3.history['val_accuracy'])
plt.title('model3 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
Around epoch 20, the validation accuracy is highest. The model seems to be performing decently around epoch 22. There is some overfitting in the data.
Evaluation of model3 model.evaluate using the Test data
# Evaluate the model on the test data
score3 = model3.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score3[0])
print('Test accuracy:', score3[1])
Test loss: 0.9419645071029663 Test accuracy: 0.75789475440979
# Test Prediction
y_pred = model3.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)
15/15 [==============================] - 0s 7ms/step
# Test Accuracy
accuracyScore3 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore3)
0.7578947368421053
Classification Report
# Compute the classification report
cr3 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)
print('Classification report:\n',cr3)
Classification report:
precision recall f1-score support
Small-flowered Cranesbill 0.90 0.86 0.88 50
Fat Hen 0.89 0.83 0.86 48
Shepherds Purse 0.62 0.57 0.59 23
Common wheat 0.55 0.73 0.63 22
Common Chickweed 0.90 0.92 0.91 61
Charlock 0.77 0.92 0.84 39
Cleavers 0.73 0.93 0.82 29
Scentless Mayweed 0.77 0.71 0.74 52
Sugar beet 0.74 0.68 0.71 38
Maize 0.80 0.55 0.65 22
Black-grass 0.39 0.50 0.44 26
Loose Silky-bent 0.75 0.63 0.68 65
accuracy 0.76 475
macro avg 0.73 0.74 0.73 475
weighted avg 0.77 0.76 0.76 475
Model3s overall accuracy is at 0.76, and the macro average F1-score is at 0.73, while the weighted average is at 0.7.
The model performs well on some classes, such as Small-flowered Cranesbill, Fat Hen, Common Chickweed, and Charlock, but doesn't perform well with others like Black-grass and Shepherds Purse.
Scoresdf = Scoresdf.append({'model':'Model3','accuracy':accuracyScore3,'layers':'10','total parameters':'3,306,188', 'neurons':'128', 'optimizer':'SGD'},ignore_index=True)
Confusion Matrix
# Computer the Confusion matrix
cm3 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
sns.heatmap(cm3, xticklabels=name_classes, yticklabels=name_classes, annot=True)
<Axes: >
Overall, model2 performs better than model3.
Lets try adding more layers to try to improve the previous models.
Layers : 17
Layer 1: Convolutional layer : 32 filters, Kernel_size = (3,3), Relu
Layer 2: BatchNormalization()
Layer 3: MaxPooling2D layer
Layer 4: Convolutional layer : 64 filters, Kernel_size = (3,3), Relu
Layer 5: BatchNormalization()
Layer 6: MaxPooling2D layer
Layer 7: Convolutional layer : 128 filters, Kernel_size = (3,3), Relu
Layer 8: BatchNormalization()
Layer 9: MaxPooling2D layer
Layer 7: Convolutional layer : 128 filters, Kernel_size = (3,3), Relu
Layer 8: BatchNormalization()
Layer 9: MaxPooling2D layer
Layer 10: Flatten layer
Layer 11: Dense layer : 256 neurons, Relu
Layer 12: BatchNormalization
Layer 13: Droput layer: 0.5
Optimizer: Adam
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
Model Implementation
# initialized a sequential model
model4 = Sequential()
model4.add(Conv2D(32, kernel_size=(3,3), activation='relu', input_shape=input_shape))
model4.add(BatchNormalization())
model4.add(MaxPooling2D(pool_size=(2,2)))
model4.add(Conv2D(64, kernel_size=(3,3), activation='relu'))
model4.add(BatchNormalization())
model4.add(MaxPooling2D(pool_size=(2,2)))
model4.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
model4.add(BatchNormalization())
model4.add(MaxPooling2D(pool_size=(2,2)))
model4.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
model4.add(BatchNormalization())
model4.add(MaxPooling2D(pool_size=(2,2)))
model4.add(Flatten())
model4.add(Dense(128, activation='relu'))
model4.add(BatchNormalization())
model4.add(Dropout(0.5))
model4.add(Dense(num_classes, activation='softmax'))
# Compile the model
model4.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
# Print the model summary
model4.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 126, 126, 32) 896
batch_normalization (BatchN (None, 126, 126, 32) 128
ormalization)
max_pooling2d (MaxPooling2D (None, 63, 63, 32) 0
)
conv2d_1 (Conv2D) (None, 61, 61, 64) 18496
batch_normalization_1 (Batc (None, 61, 61, 64) 256
hNormalization)
max_pooling2d_1 (MaxPooling (None, 30, 30, 64) 0
2D)
conv2d_2 (Conv2D) (None, 28, 28, 128) 73856
batch_normalization_2 (Batc (None, 28, 28, 128) 512
hNormalization)
max_pooling2d_2 (MaxPooling (None, 14, 14, 128) 0
2D)
conv2d_3 (Conv2D) (None, 12, 12, 128) 147584
batch_normalization_3 (Batc (None, 12, 12, 128) 512
hNormalization)
max_pooling2d_3 (MaxPooling (None, 6, 6, 128) 0
2D)
flatten (Flatten) (None, 4608) 0
dense (Dense) (None, 128) 589952
batch_normalization_4 (Batc (None, 128) 512
hNormalization)
dropout (Dropout) (None, 128) 0
dense_1 (Dense) (None, 12) 1548
=================================================================
Total params: 834,252
Trainable params: 833,292
Non-trainable params: 960
_________________________________________________________________
Model Visualization
#visualize model4 architecture
plot_model(model4, to_file='model4_plot.png', show_shapes=True, show_layer_names=True)
#visualize model4 architecture using visualkeras
visualkeras.layered_view(model4, legend=True)
Early Stopping
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 100 epochs and validation_split as 10%
history4=model4.fit(X_train,
y_train_onehot,
epochs=100,
batch_size=32,validation_split=0.10,callbacks=[es, mc])
Epoch 1/100 121/121 [==============================] - ETA: 0s - loss: 1.6494 - accuracy: 0.4936 Epoch 1: val_accuracy improved from -inf to 0.06308, saving model to best_model.h5 121/121 [==============================] - 7s 32ms/step - loss: 1.6494 - accuracy: 0.4936 - val_loss: 7.1415 - val_accuracy: 0.0631 Epoch 2/100 121/121 [==============================] - ETA: 0s - loss: 0.8717 - accuracy: 0.7200 Epoch 2: val_accuracy improved from 0.06308 to 0.12150, saving model to best_model.h5 121/121 [==============================] - 3s 25ms/step - loss: 0.8717 - accuracy: 0.7200 - val_loss: 10.6082 - val_accuracy: 0.1215 Epoch 3/100 121/121 [==============================] - ETA: 0s - loss: 0.6221 - accuracy: 0.7933 Epoch 3: val_accuracy did not improve from 0.12150 121/121 [==============================] - 3s 25ms/step - loss: 0.6221 - accuracy: 0.7933 - val_loss: 8.8956 - val_accuracy: 0.1215 Epoch 4/100 121/121 [==============================] - ETA: 0s - loss: 0.5206 - accuracy: 0.8235 Epoch 4: val_accuracy improved from 0.12150 to 0.16822, saving model to best_model.h5 121/121 [==============================] - 3s 26ms/step - loss: 0.5206 - accuracy: 0.8235 - val_loss: 4.8730 - val_accuracy: 0.1682 Epoch 5/100 120/121 [============================>.] - ETA: 0s - loss: 0.5026 - accuracy: 0.8242 Epoch 5: val_accuracy improved from 0.16822 to 0.43224, saving model to best_model.h5 121/121 [==============================] - 3s 28ms/step - loss: 0.5031 - accuracy: 0.8238 - val_loss: 2.1874 - val_accuracy: 0.4322 Epoch 6/100 120/121 [============================>.] - ETA: 0s - loss: 0.3585 - accuracy: 0.8747 Epoch 6: val_accuracy improved from 0.43224 to 0.61449, saving model to best_model.h5 121/121 [==============================] - 3s 26ms/step - loss: 0.3604 - accuracy: 0.8744 - val_loss: 1.2999 - val_accuracy: 0.6145 Epoch 7/100 119/121 [============================>.] - ETA: 0s - loss: 0.2995 - accuracy: 0.8984 Epoch 7: val_accuracy improved from 0.61449 to 0.72897, saving model to best_model.h5 121/121 [==============================] - 3s 26ms/step - loss: 0.2996 - accuracy: 0.8978 - val_loss: 0.7955 - val_accuracy: 0.7290 Epoch 8/100 120/121 [============================>.] - ETA: 0s - loss: 0.2362 - accuracy: 0.9227 Epoch 8: val_accuracy improved from 0.72897 to 0.86215, saving model to best_model.h5 121/121 [==============================] - 3s 26ms/step - loss: 0.2366 - accuracy: 0.9225 - val_loss: 0.4084 - val_accuracy: 0.8621 Epoch 9/100 121/121 [==============================] - ETA: 0s - loss: 0.2248 - accuracy: 0.9267 Epoch 9: val_accuracy did not improve from 0.86215 121/121 [==============================] - 3s 28ms/step - loss: 0.2248 - accuracy: 0.9267 - val_loss: 0.5949 - val_accuracy: 0.7897 Epoch 10/100 121/121 [==============================] - ETA: 0s - loss: 0.1825 - accuracy: 0.9374 Epoch 10: val_accuracy did not improve from 0.86215 121/121 [==============================] - 3s 25ms/step - loss: 0.1825 - accuracy: 0.9374 - val_loss: 0.6211 - val_accuracy: 0.7897 Epoch 11/100 119/121 [============================>.] - ETA: 0s - loss: 0.1683 - accuracy: 0.9456 Epoch 11: val_accuracy did not improve from 0.86215 121/121 [==============================] - 3s 25ms/step - loss: 0.1692 - accuracy: 0.9452 - val_loss: 0.5250 - val_accuracy: 0.8294 Epoch 12/100 119/121 [============================>.] - ETA: 0s - loss: 0.1375 - accuracy: 0.9572 Epoch 12: val_accuracy did not improve from 0.86215 121/121 [==============================] - 3s 25ms/step - loss: 0.1386 - accuracy: 0.9568 - val_loss: 0.7938 - val_accuracy: 0.7687 Epoch 13/100 120/121 [============================>.] - ETA: 0s - loss: 0.1108 - accuracy: 0.9672 Epoch 13: val_accuracy did not improve from 0.86215 121/121 [==============================] - 3s 27ms/step - loss: 0.1106 - accuracy: 0.9672 - val_loss: 1.3146 - val_accuracy: 0.6402 Epoch 14/100 120/121 [============================>.] - ETA: 0s - loss: 0.0849 - accuracy: 0.9779 Epoch 14: val_accuracy did not improve from 0.86215 121/121 [==============================] - 3s 26ms/step - loss: 0.0849 - accuracy: 0.9779 - val_loss: 0.5044 - val_accuracy: 0.8505 Epoch 15/100 121/121 [==============================] - ETA: 0s - loss: 0.0858 - accuracy: 0.9763 Epoch 15: val_accuracy did not improve from 0.86215 121/121 [==============================] - 3s 25ms/step - loss: 0.0858 - accuracy: 0.9763 - val_loss: 0.4750 - val_accuracy: 0.8621 Epoch 16/100 121/121 [==============================] - ETA: 0s - loss: 0.0791 - accuracy: 0.9776 Epoch 16: val_accuracy did not improve from 0.86215 121/121 [==============================] - 3s 26ms/step - loss: 0.0791 - accuracy: 0.9776 - val_loss: 1.3823 - val_accuracy: 0.6682 Epoch 17/100 119/121 [============================>.] - ETA: 0s - loss: 0.0698 - accuracy: 0.9785 Epoch 17: val_accuracy did not improve from 0.86215 121/121 [==============================] - 3s 27ms/step - loss: 0.0694 - accuracy: 0.9787 - val_loss: 0.9728 - val_accuracy: 0.7313 Epoch 18/100 120/121 [============================>.] - ETA: 0s - loss: 0.0666 - accuracy: 0.9836 Epoch 18: val_accuracy did not improve from 0.86215 121/121 [==============================] - 3s 26ms/step - loss: 0.0666 - accuracy: 0.9836 - val_loss: 0.4625 - val_accuracy: 0.8575 Epoch 18: early stopping
Training stopped on epoch 33. The highest validation accuracy was 0.8925 where the validation loss was 0.3573.
Model4 Accuracy vs Epoch curve
plt.plot(history4.history['accuracy'])
plt.plot(history4.history['val_accuracy'])
plt.title('model4 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
Notice that the model tends to overfit as epochs increased.
Evaluation of model4 model.evaluate using the Test data
# Evaluate the model on the test data
score4 = model4.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score4[0])
print('Test accuracy:', score4[1])
Test loss: 0.6450297832489014 Test accuracy: 0.8336842060089111
# Test Prediction
y_pred = model4.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)
15/15 [==============================] - 0s 8ms/step
# Test Accuracy
accuracyScore4 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore4)
0.8336842105263158
Classification Report
# Compute the classification report
cr4 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)
print('Classification report:\n',cr4)
Classification report:
precision recall f1-score support
Small-flowered Cranesbill 0.92 0.88 0.90 50
Fat Hen 0.91 0.88 0.89 48
Shepherds Purse 0.88 0.61 0.72 23
Common wheat 0.84 0.73 0.78 22
Common Chickweed 0.91 0.97 0.94 61
Charlock 0.81 0.97 0.88 39
Cleavers 0.58 0.97 0.73 29
Scentless Mayweed 0.94 0.90 0.92 52
Sugar beet 0.91 0.79 0.85 38
Maize 0.95 0.86 0.90 22
Black-grass 0.48 0.50 0.49 26
Loose Silky-bent 0.82 0.71 0.76 65
accuracy 0.83 475
macro avg 0.83 0.81 0.81 475
weighted avg 0.85 0.83 0.83 475
Model4 has an overall accuracy of 83%.
The precision scores range from 0.48 to 0.95, indicating that the model's ability to correctly identify positive instances varies by class. For example, the model has high precision in identifying Charlock and Cleavers, while it has relatively lower precision in identifying Black-grass and Shepherds Purse.
The recall scores range from 0.50 to 0.97, indicating that the model's ability to correctly identify positive instances also varies by class. For example, the model has high recall in identifying Common Chickweed and Charlock, while it has relatively lower recall in identifying Black-grass and Shepherds Purse.
The F1-scores are a harmonic mean of precision and recall and range from 0.49 to 0.92. The F1-score provides a balance between precision and recall, indicating the overall performance of the model for each class.
Scoresdf = Scoresdf.append({'model':'Model4','accuracy':accuracyScore4,'layers':'17','total parameters':'834,252', 'neurons':'128', 'optimizer':'Adam'},ignore_index=True)
Confusion Matrix
# Compute Confusion matrix
cm4 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
sns.heatmap(cm4, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()
Model4 seems to perform well overall, but there is some variation in the performance of the model for different classes.
To continue trying to improve the previous model, we add more layers to our model.
Layers : 19
Layer 1: Convolutional layer : 32 filters, Kernel_size = (3,3), Relu
Layer 2: BatchNormalization()
Layer 3: MaxPooling2D layer
Layer 4: Convolutional layer : 64 filters, Kernel_size = (3,3), Relu
Layer 5: BatchNormalization()
Layer 6: MaxPooling2D layer
Layer 7: Convolutional layer : 128 filters, Kernel_size = (3,3), Relu
Layer 8: BatchNormalization()
Layer 9: MaxPooling2D layer
Layer 10: Convolutional layer : 256 filters, Kernel_size = (3,3), Relu
Layer 11: BatchNormalization()
Layer 12: MaxPooling2D layer
Layer 13: Convolutional layer : 128 filters, Kernel_size = (3,3), Relu
Layer 14: BatchNormalization()
Layer 15: MaxPooling2D layer
Layer 16: Flatten layer
Layer 17: Dense layer : 256 neurons, Relu
Layer 18: BatchNormalization
Layer 19: Droput layer: 0.5
Optimizer: Adam
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
Model Implementation
# initialized a sequential model
model5 = Sequential()
model5.add(Conv2D(32, kernel_size=(3,3), activation='relu', input_shape=input_shape))
model5.add(BatchNormalization())
model5.add(MaxPooling2D(pool_size=(2,2)))
model5.add(Conv2D(64, kernel_size=(3,3), activation='relu'))
model5.add(BatchNormalization())
model5.add(MaxPooling2D(pool_size=(2,2)))
model5.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
model5.add(BatchNormalization())
model5.add(MaxPooling2D(pool_size=(2,2)))
model5.add(Conv2D(256, kernel_size=(3,3), activation='relu'))
model5.add(BatchNormalization())
model5.add(MaxPooling2D(pool_size=(2,2)))
model5.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
model5.add(BatchNormalization())
model5.add(MaxPooling2D(pool_size=(2,2)))
model5.add(Flatten())
model5.add(Dense(256, activation='relu'))
model5.add(BatchNormalization())
model5.add(Dropout(0.5))
model5.add(Dense(num_classes, activation='softmax'))
# Compile the model
model5.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
# Print the model summary
model5.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 126, 126, 32) 896
batch_normalization (BatchN (None, 126, 126, 32) 128
ormalization)
max_pooling2d (MaxPooling2D (None, 63, 63, 32) 0
)
conv2d_1 (Conv2D) (None, 61, 61, 64) 18496
batch_normalization_1 (Batc (None, 61, 61, 64) 256
hNormalization)
max_pooling2d_1 (MaxPooling (None, 30, 30, 64) 0
2D)
conv2d_2 (Conv2D) (None, 28, 28, 128) 73856
batch_normalization_2 (Batc (None, 28, 28, 128) 512
hNormalization)
max_pooling2d_2 (MaxPooling (None, 14, 14, 128) 0
2D)
conv2d_3 (Conv2D) (None, 12, 12, 256) 295168
batch_normalization_3 (Batc (None, 12, 12, 256) 1024
hNormalization)
max_pooling2d_3 (MaxPooling (None, 6, 6, 256) 0
2D)
conv2d_4 (Conv2D) (None, 4, 4, 128) 295040
batch_normalization_4 (Batc (None, 4, 4, 128) 512
hNormalization)
max_pooling2d_4 (MaxPooling (None, 2, 2, 128) 0
2D)
flatten (Flatten) (None, 512) 0
dense (Dense) (None, 256) 131328
batch_normalization_5 (Batc (None, 256) 1024
hNormalization)
dropout (Dropout) (None, 256) 0
dense_1 (Dense) (None, 12) 3084
=================================================================
Total params: 821,324
Trainable params: 819,596
Non-trainable params: 1,728
_________________________________________________________________
Model5 Architecture Visualization
##visualize model architecture
plot_model(model5, to_file='model5_plot.png', show_shapes=True, show_layer_names=True)
##model architecture visualization using visualkeras
visualkeras.layered_view(model5, legend=True)
Early Stopping
Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=5)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 100 epochs and validation_split as 10%
history5 = model5.fit(X_train,
y_train_onehot,
epochs=100,
batch_size=32,validation_split=0.10,callbacks=[es, mc])
Epoch 1/100 121/121 [==============================] - ETA: 0s - loss: 1.6742 - accuracy: 0.5027 Epoch 1: val_accuracy improved from -inf to 0.06308, saving model to best_model.h5 121/121 [==============================] - 15s 41ms/step - loss: 1.6742 - accuracy: 0.5027 - val_loss: 7.5660 - val_accuracy: 0.0631 Epoch 2/100 121/121 [==============================] - ETA: 0s - loss: 0.9033 - accuracy: 0.6964 Epoch 2: val_accuracy improved from 0.06308 to 0.12150, saving model to best_model.h5 121/121 [==============================] - 3s 28ms/step - loss: 0.9033 - accuracy: 0.6964 - val_loss: 14.3085 - val_accuracy: 0.1215 Epoch 3/100 119/121 [============================>.] - ETA: 0s - loss: 0.6271 - accuracy: 0.7844 Epoch 3: val_accuracy did not improve from 0.12150 121/121 [==============================] - 3s 26ms/step - loss: 0.6265 - accuracy: 0.7840 - val_loss: 11.6241 - val_accuracy: 0.1215 Epoch 4/100 119/121 [============================>.] - ETA: 0s - loss: 0.4865 - accuracy: 0.8304 Epoch 4: val_accuracy improved from 0.12150 to 0.16822, saving model to best_model.h5 121/121 [==============================] - 3s 28ms/step - loss: 0.4887 - accuracy: 0.8300 - val_loss: 6.3291 - val_accuracy: 0.1682 Epoch 5/100 121/121 [==============================] - ETA: 0s - loss: 0.4014 - accuracy: 0.8596 Epoch 5: val_accuracy improved from 0.16822 to 0.55140, saving model to best_model.h5 121/121 [==============================] - 3s 28ms/step - loss: 0.4014 - accuracy: 0.8596 - val_loss: 1.5898 - val_accuracy: 0.5514 Epoch 6/100 120/121 [============================>.] - ETA: 0s - loss: 0.3551 - accuracy: 0.8745 Epoch 6: val_accuracy improved from 0.55140 to 0.61682, saving model to best_model.h5 121/121 [==============================] - 3s 28ms/step - loss: 0.3566 - accuracy: 0.8737 - val_loss: 1.2139 - val_accuracy: 0.6168 Epoch 7/100 119/121 [============================>.] - ETA: 0s - loss: 0.3083 - accuracy: 0.8824 Epoch 7: val_accuracy improved from 0.61682 to 0.84346, saving model to best_model.h5 121/121 [==============================] - 3s 28ms/step - loss: 0.3062 - accuracy: 0.8835 - val_loss: 0.4655 - val_accuracy: 0.8435 Epoch 8/100 120/121 [============================>.] - ETA: 0s - loss: 0.2573 - accuracy: 0.9055 Epoch 8: val_accuracy did not improve from 0.84346 121/121 [==============================] - 3s 28ms/step - loss: 0.2590 - accuracy: 0.9051 - val_loss: 1.4169 - val_accuracy: 0.5888 Epoch 9/100 121/121 [==============================] - ETA: 0s - loss: 0.2403 - accuracy: 0.9124 Epoch 9: val_accuracy did not improve from 0.84346 121/121 [==============================] - 3s 27ms/step - loss: 0.2403 - accuracy: 0.9124 - val_loss: 2.1576 - val_accuracy: 0.5210 Epoch 10/100 119/121 [============================>.] - ETA: 0s - loss: 0.1973 - accuracy: 0.9270 Epoch 10: val_accuracy improved from 0.84346 to 0.85748, saving model to best_model.h5 121/121 [==============================] - 3s 27ms/step - loss: 0.2010 - accuracy: 0.9251 - val_loss: 0.4383 - val_accuracy: 0.8575 Epoch 11/100 120/121 [============================>.] - ETA: 0s - loss: 0.1920 - accuracy: 0.9289 Epoch 11: val_accuracy did not improve from 0.85748 121/121 [==============================] - 3s 26ms/step - loss: 0.1920 - accuracy: 0.9288 - val_loss: 0.9245 - val_accuracy: 0.7336 Epoch 12/100 119/121 [============================>.] - ETA: 0s - loss: 0.1423 - accuracy: 0.9459 Epoch 12: val_accuracy did not improve from 0.85748 121/121 [==============================] - 4s 32ms/step - loss: 0.1437 - accuracy: 0.9454 - val_loss: 1.2745 - val_accuracy: 0.6916 Epoch 13/100 120/121 [============================>.] - ETA: 0s - loss: 0.2415 - accuracy: 0.9128 Epoch 13: val_accuracy did not improve from 0.85748 121/121 [==============================] - 3s 27ms/step - loss: 0.2430 - accuracy: 0.9124 - val_loss: 3.1072 - val_accuracy: 0.3178 Epoch 14/100 120/121 [============================>.] - ETA: 0s - loss: 0.1829 - accuracy: 0.9331 Epoch 14: val_accuracy did not improve from 0.85748 121/121 [==============================] - 3s 26ms/step - loss: 0.1833 - accuracy: 0.9327 - val_loss: 1.6813 - val_accuracy: 0.5771 Epoch 15/100 121/121 [==============================] - ETA: 0s - loss: 0.1277 - accuracy: 0.9563 Epoch 15: val_accuracy did not improve from 0.85748 121/121 [==============================] - 3s 26ms/step - loss: 0.1277 - accuracy: 0.9563 - val_loss: 0.5376 - val_accuracy: 0.8294 Epoch 15: early stopping
Training stopped at epoch 15. The highest validation accuracy was 0.8575 and validation loss is 0.4383.
Accuracy vs Epoch curve
plt.plot(history5.history['accuracy'])
plt.plot(history5.history['val_accuracy'])
plt.title('model5 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
The model is overfitting. However, around epoch 9, it has a high validation accuracy and a high training accuracy.
Evaluation of model5 model.evaluate using the Test data
# Evaluate the model on the test data
score5 = model5.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score5[0])
print('Test accuracy:', score5[1])
Test loss: 0.5987040400505066 Test accuracy: 0.8252631425857544
# Test Prediction
y_pred = model5.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)
15/15 [==============================] - 0s 9ms/step
# Test Accuracy
accuracyScore5 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore5)
0.8252631578947368
Classification Report
# Compute the classification report
cr5 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)
print('Classification report:\n',cr5)
Classification report:
precision recall f1-score support
Small-flowered Cranesbill 0.90 0.94 0.92 50
Fat Hen 1.00 0.83 0.91 48
Shepherds Purse 0.90 0.78 0.84 23
Common wheat 0.80 0.55 0.65 22
Common Chickweed 0.88 0.92 0.90 61
Charlock 0.95 1.00 0.97 39
Cleavers 0.90 0.90 0.90 29
Scentless Mayweed 0.79 0.96 0.87 52
Sugar beet 0.95 0.55 0.70 38
Maize 0.94 0.73 0.82 22
Black-grass 0.39 0.27 0.32 26
Loose Silky-bent 0.64 0.92 0.75 65
accuracy 0.83 475
macro avg 0.84 0.78 0.80 475
weighted avg 0.84 0.83 0.82 475
The accuracy of the model5 is 83%, and the weighted average F1-score is 0.82. The model performs relatively well in predicting most of the plant species, with high precision, recall and f1-scores, except for Black-grass.
However Model4 seems to be performing slightly better.
Scoresdf = Scoresdf.append({'model':'Model5','accuracy':accuracyScore5,'layers':'20','total parameters':'821,324', 'neurons':'256', 'optimizer':'Adam'},ignore_index=True)
Confusion Matrix
#Visualize the Confusion matrix
cm5 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
sns.heatmap(cm5, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()
Model5 performs well for most classes but Black-grass.
To try to improve the performance of our previous models, we changethe layers and filters.
Layers: 16
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
Model Implementation
# initialized a sequential model
model6 = Sequential()
model6.add(Conv2D(64, kernel_size=(3,3), activation='relu', input_shape=input_shape))
model6.add(BatchNormalization())
model6.add(MaxPooling2D(pool_size=(2,2)))
model6.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
model6.add(BatchNormalization())
model6.add(MaxPooling2D(pool_size=(2,2)))
model6.add(Conv2D(256, kernel_size=(3,3), activation='relu'))
model6.add(BatchNormalization())
model6.add(MaxPooling2D(pool_size=(2,2)))
model6.add(Flatten())
model6.add(Dense(512, activation='relu'))
model6.add(BatchNormalization())
model6.add(Dropout(0.5))
model6.add(Dense(256, activation='relu'))
model6.add(BatchNormalization())
model6.add(Dropout(0.5))
model6.add(Dense(num_classes, activation='softmax'))
# Compile the model
model6.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
# Print the model summary
model6.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 126, 126, 64) 1792
batch_normalization (BatchN (None, 126, 126, 64) 256
ormalization)
max_pooling2d (MaxPooling2D (None, 63, 63, 64) 0
)
conv2d_1 (Conv2D) (None, 61, 61, 128) 73856
batch_normalization_1 (Batc (None, 61, 61, 128) 512
hNormalization)
max_pooling2d_1 (MaxPooling (None, 30, 30, 128) 0
2D)
conv2d_2 (Conv2D) (None, 28, 28, 256) 295168
batch_normalization_2 (Batc (None, 28, 28, 256) 1024
hNormalization)
max_pooling2d_2 (MaxPooling (None, 14, 14, 256) 0
2D)
flatten (Flatten) (None, 50176) 0
dense (Dense) (None, 512) 25690624
batch_normalization_3 (Batc (None, 512) 2048
hNormalization)
dropout (Dropout) (None, 512) 0
dense_1 (Dense) (None, 256) 131328
batch_normalization_4 (Batc (None, 256) 1024
hNormalization)
dropout_1 (Dropout) (None, 256) 0
dense_2 (Dense) (None, 12) 3084
=================================================================
Total params: 26,200,716
Trainable params: 26,198,284
Non-trainable params: 2,432
_________________________________________________________________
Model6 Architecture Visualization
#visualize model6 architecture
plot_model(model6, to_file='model6_plot.png', show_shapes=True, show_layer_names=True)
#visualize model6 architecture using visualkeras
visualkeras.layered_view(model6, legend=True)
Early Stopping
Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 40 epochs and validation_split as 10%
history6=model6.fit(X_train,
y_train_onehot,
epochs=40,
batch_size=32,validation_split=0.10,callbacks=[es, mc])
Epoch 1/40 121/121 [==============================] - ETA: 0s - loss: 1.9508 - accuracy: 0.4333 Epoch 1: val_accuracy improved from -inf to 0.12150, saving model to best_model.h5 121/121 [==============================] - 14s 76ms/step - loss: 1.9508 - accuracy: 0.4333 - val_loss: 7.4120 - val_accuracy: 0.1215 Epoch 2/40 120/121 [============================>.] - ETA: 0s - loss: 1.0981 - accuracy: 0.6617 Epoch 2: val_accuracy did not improve from 0.12150 121/121 [==============================] - 7s 59ms/step - loss: 1.0994 - accuracy: 0.6613 - val_loss: 8.0671 - val_accuracy: 0.1215 Epoch 3/40 120/121 [============================>.] - ETA: 0s - loss: 0.7742 - accuracy: 0.7349 Epoch 3: val_accuracy improved from 0.12150 to 0.17056, saving model to best_model.h5 121/121 [==============================] - 8s 62ms/step - loss: 0.7752 - accuracy: 0.7349 - val_loss: 7.4708 - val_accuracy: 0.1706 Epoch 4/40 120/121 [============================>.] - ETA: 0s - loss: 0.6962 - accuracy: 0.7693 Epoch 4: val_accuracy improved from 0.17056 to 0.20561, saving model to best_model.h5 121/121 [==============================] - 8s 63ms/step - loss: 0.6970 - accuracy: 0.7689 - val_loss: 4.7459 - val_accuracy: 0.2056 Epoch 5/40 121/121 [==============================] - ETA: 0s - loss: 0.4953 - accuracy: 0.8264 Epoch 5: val_accuracy improved from 0.20561 to 0.28037, saving model to best_model.h5 121/121 [==============================] - 7s 59ms/step - loss: 0.4953 - accuracy: 0.8264 - val_loss: 4.4107 - val_accuracy: 0.2804 Epoch 6/40 120/121 [============================>.] - ETA: 0s - loss: 0.4517 - accuracy: 0.8424 Epoch 6: val_accuracy improved from 0.28037 to 0.60047, saving model to best_model.h5 121/121 [==============================] - 8s 68ms/step - loss: 0.4548 - accuracy: 0.8417 - val_loss: 1.2011 - val_accuracy: 0.6005 Epoch 7/40 121/121 [==============================] - ETA: 0s - loss: 0.3429 - accuracy: 0.8815 Epoch 7: val_accuracy did not improve from 0.60047 121/121 [==============================] - 6s 52ms/step - loss: 0.3429 - accuracy: 0.8815 - val_loss: 1.6667 - val_accuracy: 0.5607 Epoch 8/40 120/121 [============================>.] - ETA: 0s - loss: 0.3029 - accuracy: 0.8940 Epoch 8: val_accuracy did not improve from 0.60047 121/121 [==============================] - 7s 58ms/step - loss: 0.3046 - accuracy: 0.8934 - val_loss: 3.8309 - val_accuracy: 0.2967 Epoch 9/40 120/121 [============================>.] - ETA: 0s - loss: 0.2561 - accuracy: 0.9104 Epoch 9: val_accuracy improved from 0.60047 to 0.62850, saving model to best_model.h5 121/121 [==============================] - 8s 66ms/step - loss: 0.2558 - accuracy: 0.9106 - val_loss: 1.3518 - val_accuracy: 0.6285 Epoch 10/40 120/121 [============================>.] - ETA: 0s - loss: 0.2026 - accuracy: 0.9318 Epoch 10: val_accuracy improved from 0.62850 to 0.80607, saving model to best_model.h5 121/121 [==============================] - 8s 64ms/step - loss: 0.2046 - accuracy: 0.9311 - val_loss: 0.7063 - val_accuracy: 0.8061 Epoch 11/40 121/121 [==============================] - ETA: 0s - loss: 0.2050 - accuracy: 0.9264 Epoch 11: val_accuracy did not improve from 0.80607 121/121 [==============================] - 6s 52ms/step - loss: 0.2050 - accuracy: 0.9264 - val_loss: 0.7386 - val_accuracy: 0.7687 Epoch 12/40 120/121 [============================>.] - ETA: 0s - loss: 0.1418 - accuracy: 0.9513 Epoch 12: val_accuracy improved from 0.80607 to 0.81308, saving model to best_model.h5 121/121 [==============================] - 8s 67ms/step - loss: 0.1422 - accuracy: 0.9511 - val_loss: 0.6383 - val_accuracy: 0.8131 Epoch 13/40 120/121 [============================>.] - ETA: 0s - loss: 0.1396 - accuracy: 0.9557 Epoch 13: val_accuracy improved from 0.81308 to 0.82243, saving model to best_model.h5 121/121 [==============================] - 7s 62ms/step - loss: 0.1396 - accuracy: 0.9558 - val_loss: 0.6111 - val_accuracy: 0.8224 Epoch 14/40 120/121 [============================>.] - ETA: 0s - loss: 0.0992 - accuracy: 0.9677 Epoch 14: val_accuracy improved from 0.82243 to 0.84813, saving model to best_model.h5 121/121 [==============================] - 8s 63ms/step - loss: 0.0992 - accuracy: 0.9678 - val_loss: 0.5729 - val_accuracy: 0.8481 Epoch 15/40 120/121 [============================>.] - ETA: 0s - loss: 0.0804 - accuracy: 0.9716 Epoch 15: val_accuracy did not improve from 0.84813 121/121 [==============================] - 6s 51ms/step - loss: 0.0820 - accuracy: 0.9714 - val_loss: 0.6829 - val_accuracy: 0.7944 Epoch 16/40 120/121 [============================>.] - ETA: 0s - loss: 0.1204 - accuracy: 0.9602 Epoch 16: val_accuracy did not improve from 0.84813 121/121 [==============================] - 6s 50ms/step - loss: 0.1204 - accuracy: 0.9602 - val_loss: 4.0306 - val_accuracy: 0.3879 Epoch 17/40 120/121 [============================>.] - ETA: 0s - loss: 0.0859 - accuracy: 0.9724 Epoch 17: val_accuracy did not improve from 0.84813 121/121 [==============================] - 6s 50ms/step - loss: 0.0858 - accuracy: 0.9724 - val_loss: 0.8944 - val_accuracy: 0.7407 Epoch 18/40 121/121 [==============================] - ETA: 0s - loss: 0.0702 - accuracy: 0.9789 Epoch 18: val_accuracy did not improve from 0.84813 121/121 [==============================] - 6s 51ms/step - loss: 0.0702 - accuracy: 0.9789 - val_loss: 1.3378 - val_accuracy: 0.6822 Epoch 19/40 120/121 [============================>.] - ETA: 0s - loss: 0.0712 - accuracy: 0.9768 Epoch 19: val_accuracy did not improve from 0.84813 121/121 [==============================] - 6s 50ms/step - loss: 0.0712 - accuracy: 0.9769 - val_loss: 1.1351 - val_accuracy: 0.7266 Epoch 20/40 120/121 [============================>.] - ETA: 0s - loss: 0.0670 - accuracy: 0.9786 Epoch 20: val_accuracy did not improve from 0.84813 121/121 [==============================] - 6s 50ms/step - loss: 0.0678 - accuracy: 0.9782 - val_loss: 2.5232 - val_accuracy: 0.4883 Epoch 21/40 120/121 [============================>.] - ETA: 0s - loss: 0.1028 - accuracy: 0.9633 Epoch 21: val_accuracy did not improve from 0.84813 121/121 [==============================] - 6s 48ms/step - loss: 0.1039 - accuracy: 0.9631 - val_loss: 0.8581 - val_accuracy: 0.7850 Epoch 22/40 120/121 [============================>.] - ETA: 0s - loss: 0.1009 - accuracy: 0.9661 Epoch 22: val_accuracy did not improve from 0.84813 121/121 [==============================] - 6s 50ms/step - loss: 0.1008 - accuracy: 0.9662 - val_loss: 1.1921 - val_accuracy: 0.7033 Epoch 23/40 120/121 [============================>.] - ETA: 0s - loss: 0.0571 - accuracy: 0.9794 Epoch 23: val_accuracy did not improve from 0.84813 121/121 [==============================] - 6s 50ms/step - loss: 0.0580 - accuracy: 0.9792 - val_loss: 0.6220 - val_accuracy: 0.8318 Epoch 24/40 121/121 [==============================] - ETA: 0s - loss: 0.0501 - accuracy: 0.9834 Epoch 24: val_accuracy did not improve from 0.84813 121/121 [==============================] - 6s 51ms/step - loss: 0.0501 - accuracy: 0.9834 - val_loss: 2.5063 - val_accuracy: 0.5818 Epoch 24: early stopping
Training stopped at epoch 24. The highest Validation accuracy was 0.8481 and validation loss 0.5729.
Model6 Accuracy vs Epoch curve
plt.plot(history6.history['accuracy'])
plt.plot(history6.history['val_accuracy'])
plt.title('model6 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
This model seems to be overfitting.
Evaluation of model6 model.evaluate using the Test data
# Evaluate the model on the test data
score6 = model6.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score6[0])
print('Test accuracy:', score6[1])
Test loss: 2.657069444656372 Test accuracy: 0.5621052384376526
# Test Prediction
y_pred = model6.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)
15/15 [==============================] - 0s 14ms/step
# Test Accuracy
accuracyScore6 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore6)
0.5621052631578948
Classification Report
# Compute the classification report
cr6 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)
print('Classification report:\n',cr6)
Classification report:
precision recall f1-score support
Small-flowered Cranesbill 0.92 0.66 0.77 50
Fat Hen 0.90 0.54 0.68 48
Shepherds Purse 0.56 0.22 0.31 23
Common wheat 0.78 0.64 0.70 22
Common Chickweed 0.95 0.33 0.49 61
Charlock 1.00 0.51 0.68 39
Cleavers 0.95 0.69 0.80 29
Scentless Mayweed 0.26 1.00 0.42 52
Sugar beet 1.00 0.03 0.05 38
Maize 0.88 0.64 0.74 22
Black-grass 0.75 0.12 0.20 26
Loose Silky-bent 0.57 0.91 0.70 65
accuracy 0.56 475
macro avg 0.79 0.52 0.54 475
weighted avg 0.78 0.56 0.55 475
The model seems to be overfitting as it has high accuracy on the training set but low accuracy on the test set. The precision, recall, and F1-score for each class also vary widely, with some classes having high scores and others having low scores. This suggests that the model is not performing well in some classes.
Scoresdf = Scoresdf.append({'model':'Model6','accuracy':accuracyScore6,'layers':'17','total parameters':'26,200,716', 'neurons':'256', 'optimizer':'Adam'},ignore_index=True)
Confusion Matrix
# Calculate Confusion matrix
cm6 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
sns.heatmap(cm5, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()
Overall, Model6 doesn't perform well for some classes and is overfitting.
To continue trying to improve our model, we added more layers and modified the filter sizes.
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
Model Implementation
#Model implementation
model7 = Sequential()
model7.add(Conv2D(32, (3, 3), activation='relu', input_shape=input_shape))
model7.add(Conv2D(32, (3, 3), activation='relu'))
model7.add(BatchNormalization())
model7.add(MaxPooling2D(pool_size=(2, 2)))
model7.add(Dropout(0.25))
model7.add(Conv2D(64, (3, 3), activation='relu'))
model7.add(Conv2D(64, (3, 3), activation='relu'))
model7.add(BatchNormalization())
model7.add(MaxPooling2D(pool_size=(2, 2)))
model7.add(Dropout(0.25))
model7.add(Conv2D(128, (3, 3), activation='relu'))
model7.add(Conv2D(128, (3, 3), activation='relu'))
model7.add(BatchNormalization())
model7.add(MaxPooling2D(pool_size=(2, 2)))
model7.add(Dropout(0.25))
model7.add(Flatten())
model7.add(Dense(512, activation='relu'))
model7.add(BatchNormalization())
model7.add(Dropout(0.5))
model7.add(Dense(num_classes, activation='softmax'))
model7.compile(loss='categorical_crossentropy',
optimizer='adam',
metrics=['accuracy'])
model7.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 126, 126, 32) 896
conv2d_1 (Conv2D) (None, 124, 124, 32) 9248
batch_normalization (BatchN (None, 124, 124, 32) 128
ormalization)
max_pooling2d (MaxPooling2D (None, 62, 62, 32) 0
)
dropout (Dropout) (None, 62, 62, 32) 0
conv2d_2 (Conv2D) (None, 60, 60, 64) 18496
conv2d_3 (Conv2D) (None, 58, 58, 64) 36928
batch_normalization_1 (Batc (None, 58, 58, 64) 256
hNormalization)
max_pooling2d_1 (MaxPooling (None, 29, 29, 64) 0
2D)
dropout_1 (Dropout) (None, 29, 29, 64) 0
conv2d_4 (Conv2D) (None, 27, 27, 128) 73856
conv2d_5 (Conv2D) (None, 25, 25, 128) 147584
batch_normalization_2 (Batc (None, 25, 25, 128) 512
hNormalization)
max_pooling2d_2 (MaxPooling (None, 12, 12, 128) 0
2D)
dropout_2 (Dropout) (None, 12, 12, 128) 0
flatten (Flatten) (None, 18432) 0
dense (Dense) (None, 512) 9437696
batch_normalization_3 (Batc (None, 512) 2048
hNormalization)
dropout_3 (Dropout) (None, 512) 0
dense_1 (Dense) (None, 12) 6156
=================================================================
Total params: 9,733,804
Trainable params: 9,732,332
Non-trainable params: 1,472
_________________________________________________________________
Model7 Architecture Visualization
#visualize model7 architecture
plot_model(model7, to_file='model7_plot.png', show_shapes=True, show_layer_names=True)
#visualize model7 architecture using visualkeras
visualkeras.layered_view(model7, legend=True)
Early Stopping
Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 100 epochs and validation_split as 10%
history7=model7.fit(X_train,
y_train_onehot,
epochs=100,
batch_size=32,validation_split=0.10,callbacks=[es, mc])
Epoch 1/100 121/121 [==============================] - ETA: 0s - loss: 1.8702 - accuracy: 0.4645 Epoch 1: val_accuracy improved from -inf to 0.12150, saving model to best_model.h5 121/121 [==============================] - 15s 77ms/step - loss: 1.8702 - accuracy: 0.4645 - val_loss: 16.9230 - val_accuracy: 0.1215 Epoch 2/100 120/121 [============================>.] - ETA: 0s - loss: 0.9498 - accuracy: 0.7016 Epoch 2: val_accuracy did not improve from 0.12150 121/121 [==============================] - 8s 69ms/step - loss: 0.9510 - accuracy: 0.7011 - val_loss: 17.2805 - val_accuracy: 0.1215 Epoch 3/100 120/121 [============================>.] - ETA: 0s - loss: 0.6415 - accuracy: 0.7870 Epoch 3: val_accuracy did not improve from 0.12150 121/121 [==============================] - 8s 64ms/step - loss: 0.6428 - accuracy: 0.7868 - val_loss: 17.0637 - val_accuracy: 0.1215 Epoch 4/100 120/121 [============================>.] - ETA: 0s - loss: 0.5017 - accuracy: 0.8240 Epoch 4: val_accuracy improved from 0.12150 to 0.14019, saving model to best_model.h5 121/121 [==============================] - 8s 66ms/step - loss: 0.5030 - accuracy: 0.8235 - val_loss: 8.0797 - val_accuracy: 0.1402 Epoch 5/100 120/121 [============================>.] - ETA: 0s - loss: 0.4119 - accuracy: 0.8542 Epoch 5: val_accuracy improved from 0.14019 to 0.34579, saving model to best_model.h5 121/121 [==============================] - 8s 67ms/step - loss: 0.4144 - accuracy: 0.8531 - val_loss: 4.5227 - val_accuracy: 0.3458 Epoch 6/100 120/121 [============================>.] - ETA: 0s - loss: 0.3486 - accuracy: 0.8805 Epoch 6: val_accuracy improved from 0.34579 to 0.50467, saving model to best_model.h5 121/121 [==============================] - 8s 67ms/step - loss: 0.3540 - accuracy: 0.8794 - val_loss: 1.6728 - val_accuracy: 0.5047 Epoch 7/100 120/121 [============================>.] - ETA: 0s - loss: 0.2966 - accuracy: 0.8930 Epoch 7: val_accuracy improved from 0.50467 to 0.53271, saving model to best_model.h5 121/121 [==============================] - 8s 67ms/step - loss: 0.2981 - accuracy: 0.8926 - val_loss: 2.7445 - val_accuracy: 0.5327 Epoch 8/100 120/121 [============================>.] - ETA: 0s - loss: 0.2273 - accuracy: 0.9174 Epoch 8: val_accuracy improved from 0.53271 to 0.60047, saving model to best_model.h5 121/121 [==============================] - 8s 65ms/step - loss: 0.2295 - accuracy: 0.9168 - val_loss: 1.3810 - val_accuracy: 0.6005 Epoch 9/100 120/121 [============================>.] - ETA: 0s - loss: 0.1840 - accuracy: 0.9326 Epoch 9: val_accuracy improved from 0.60047 to 0.80841, saving model to best_model.h5 121/121 [==============================] - 8s 67ms/step - loss: 0.1838 - accuracy: 0.9327 - val_loss: 0.7627 - val_accuracy: 0.8084 Epoch 10/100 120/121 [============================>.] - ETA: 0s - loss: 0.1397 - accuracy: 0.9518 Epoch 10: val_accuracy did not improve from 0.80841 121/121 [==============================] - 8s 63ms/step - loss: 0.1416 - accuracy: 0.9511 - val_loss: 2.6548 - val_accuracy: 0.4206 Epoch 11/100 120/121 [============================>.] - ETA: 0s - loss: 0.1373 - accuracy: 0.9492 Epoch 11: val_accuracy did not improve from 0.80841 121/121 [==============================] - 8s 65ms/step - loss: 0.1371 - accuracy: 0.9493 - val_loss: 0.7245 - val_accuracy: 0.7757 Epoch 12/100 120/121 [============================>.] - ETA: 0s - loss: 0.0871 - accuracy: 0.9729 Epoch 12: val_accuracy did not improve from 0.80841 121/121 [==============================] - 8s 66ms/step - loss: 0.0877 - accuracy: 0.9724 - val_loss: 2.6663 - val_accuracy: 0.5981 Epoch 13/100 120/121 [============================>.] - ETA: 0s - loss: 0.1169 - accuracy: 0.9591 Epoch 13: val_accuracy did not improve from 0.80841 121/121 [==============================] - 8s 66ms/step - loss: 0.1168 - accuracy: 0.9592 - val_loss: 1.3031 - val_accuracy: 0.7547 Epoch 14/100 120/121 [============================>.] - ETA: 0s - loss: 0.0866 - accuracy: 0.9714 Epoch 14: val_accuracy did not improve from 0.80841 121/121 [==============================] - 8s 65ms/step - loss: 0.0873 - accuracy: 0.9711 - val_loss: 1.3552 - val_accuracy: 0.7079 Epoch 15/100 120/121 [============================>.] - ETA: 0s - loss: 0.0901 - accuracy: 0.9703 Epoch 15: val_accuracy did not improve from 0.80841 121/121 [==============================] - 8s 66ms/step - loss: 0.0925 - accuracy: 0.9701 - val_loss: 1.2023 - val_accuracy: 0.7313 Epoch 16/100 120/121 [============================>.] - ETA: 0s - loss: 0.0702 - accuracy: 0.9771 Epoch 16: val_accuracy improved from 0.80841 to 0.81308, saving model to best_model.h5 121/121 [==============================] - 8s 67ms/step - loss: 0.0701 - accuracy: 0.9771 - val_loss: 0.8398 - val_accuracy: 0.8131 Epoch 17/100 120/121 [============================>.] - ETA: 0s - loss: 0.0457 - accuracy: 0.9859 Epoch 17: val_accuracy improved from 0.81308 to 0.85748, saving model to best_model.h5 121/121 [==============================] - 9s 72ms/step - loss: 0.0458 - accuracy: 0.9860 - val_loss: 0.5525 - val_accuracy: 0.8575 Epoch 18/100 120/121 [============================>.] - ETA: 0s - loss: 0.0554 - accuracy: 0.9846 Epoch 18: val_accuracy did not improve from 0.85748 121/121 [==============================] - 8s 67ms/step - loss: 0.0554 - accuracy: 0.9847 - val_loss: 0.6726 - val_accuracy: 0.8294 Epoch 19/100 120/121 [============================>.] - ETA: 0s - loss: 0.0684 - accuracy: 0.9768 Epoch 19: val_accuracy did not improve from 0.85748 121/121 [==============================] - 8s 65ms/step - loss: 0.0697 - accuracy: 0.9766 - val_loss: 0.7490 - val_accuracy: 0.8154 Epoch 20/100 120/121 [============================>.] - ETA: 0s - loss: 0.1246 - accuracy: 0.9576 Epoch 20: val_accuracy did not improve from 0.85748 121/121 [==============================] - 7s 62ms/step - loss: 0.1257 - accuracy: 0.9568 - val_loss: 1.4332 - val_accuracy: 0.6729 Epoch 21/100 120/121 [============================>.] - ETA: 0s - loss: 0.1788 - accuracy: 0.9367 Epoch 21: val_accuracy did not improve from 0.85748 121/121 [==============================] - 8s 65ms/step - loss: 0.1787 - accuracy: 0.9366 - val_loss: 7.4364 - val_accuracy: 0.2897 Epoch 22/100 120/121 [============================>.] - ETA: 0s - loss: 0.0974 - accuracy: 0.9685 Epoch 22: val_accuracy did not improve from 0.85748 121/121 [==============================] - 8s 65ms/step - loss: 0.0973 - accuracy: 0.9685 - val_loss: 2.1218 - val_accuracy: 0.6519 Epoch 23/100 120/121 [============================>.] - ETA: 0s - loss: 0.0734 - accuracy: 0.9740 Epoch 23: val_accuracy did not improve from 0.85748 121/121 [==============================] - 8s 66ms/step - loss: 0.0745 - accuracy: 0.9737 - val_loss: 2.0879 - val_accuracy: 0.5654 Epoch 24/100 120/121 [============================>.] - ETA: 0s - loss: 0.0601 - accuracy: 0.9776 Epoch 24: val_accuracy did not improve from 0.85748 121/121 [==============================] - 8s 64ms/step - loss: 0.0600 - accuracy: 0.9776 - val_loss: 1.2884 - val_accuracy: 0.6939 Epoch 25/100 120/121 [============================>.] - ETA: 0s - loss: 0.0553 - accuracy: 0.9836 Epoch 25: val_accuracy did not improve from 0.85748 121/121 [==============================] - 8s 64ms/step - loss: 0.0562 - accuracy: 0.9834 - val_loss: 0.9433 - val_accuracy: 0.7757 Epoch 26/100 120/121 [============================>.] - ETA: 0s - loss: 0.0517 - accuracy: 0.9852 Epoch 26: val_accuracy did not improve from 0.85748 121/121 [==============================] - 8s 69ms/step - loss: 0.0534 - accuracy: 0.9849 - val_loss: 0.7878 - val_accuracy: 0.8435 Epoch 27/100 120/121 [============================>.] - ETA: 0s - loss: 0.0514 - accuracy: 0.9852 Epoch 27: val_accuracy did not improve from 0.85748 121/121 [==============================] - 8s 65ms/step - loss: 0.0523 - accuracy: 0.9849 - val_loss: 9.2228 - val_accuracy: 0.2547 Epoch 27: early stopping
Training stopped at epoch 27. The highest validation accuracy was 0.8578 with validation loss of 0.25
Model7 Accuracy vs Epoch curve
plt.plot(history7.history['accuracy'])
plt.plot(history7.history['val_accuracy'])
plt.title('model7 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
Model is overfitting
Model7 Evaluation of model model.evaluate using the Test data
# Evaluate the model on the test data
score7 = model7.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score7[0])
print('Test accuracy:', score7[1])
Test loss: 9.582252502441406 Test accuracy: 0.24421052634716034
# Test Prediction
y_pred = model7.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)
15/15 [==============================] - 0s 14ms/step
# Test Accuracy
accuracyScore7 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore7)
0.24421052631578946
Classification Report
# Compute the classification report
cr7 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)
print('Classification report:\n',cr7)
Classification report:
precision recall f1-score support
Small-flowered Cranesbill 0.95 0.36 0.52 50
Fat Hen 0.00 0.00 0.00 48
Shepherds Purse 0.00 0.00 0.00 23
Common wheat 0.17 0.55 0.26 22
Common Chickweed 0.00 0.00 0.00 61
Charlock 0.28 0.90 0.43 39
Cleavers 0.00 0.00 0.00 29
Scentless Mayweed 0.22 0.67 0.33 52
Sugar beet 0.00 0.00 0.00 38
Maize 0.12 0.50 0.19 22
Black-grass 0.00 0.00 0.00 26
Loose Silky-bent 1.00 0.08 0.14 65
accuracy 0.24 475
macro avg 0.23 0.25 0.16 475
weighted avg 0.30 0.24 0.17 475
The accuracy of the model is very low at 0.24. For most of the classes, precision, recall, and f1-score are also very low. This suggests that the model is not able to classify the images correctly, and the predictions made by the model are almost random. Therefore, it is not a valid model.
Scoresdf = Scoresdf.append({'model':'Model7','accuracy':accuracyScore7,'layers':'20','total parameters':'9,733,804', 'neurons':'512', 'optimizer':'Adam'},ignore_index=True)
Confusion Matrix
# Compute the Confusion matrix
cm7 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
sns.heatmap(cm7, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.show()
Model7 is not a valid model.
Continuing trying to improve the previous model, we changed the number of layers and neurons.
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
Model Implementation
# initialized a sequential model
model8 = Sequential()
model8.add(Conv2D(64, kernel_size=(3,3), activation='relu', input_shape=input_shape))
model8.add(BatchNormalization())
model8.add(MaxPooling2D(pool_size=(2,2)))
model8.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
model8.add(BatchNormalization())
model8.add(MaxPooling2D(pool_size=(2,2)))
model8.add(Conv2D(256, kernel_size=(3,3), activation='relu'))
model8.add(BatchNormalization())
model8.add(MaxPooling2D(pool_size=(2,2)))
model8.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
model8.add(BatchNormalization())
model8.add(MaxPooling2D(pool_size=(2,2)))
model8.add(Conv2D(64, kernel_size=(5,5), activation='relu'))
model8.add(BatchNormalization())
model8.add(MaxPooling2D(pool_size=(2,2)))
model8.add(Flatten())
model8.add(Dense(256, activation='relu'))
model8.add(BatchNormalization())
model8.add(Dropout(0.5))
model8.add(Dense(num_classes, activation='softmax'))
# Compile the model
model8.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
# Print the model summary
model8.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 126, 126, 64) 1792
batch_normalization (BatchN (None, 126, 126, 64) 256
ormalization)
max_pooling2d (MaxPooling2D (None, 63, 63, 64) 0
)
conv2d_1 (Conv2D) (None, 61, 61, 128) 73856
batch_normalization_1 (Batc (None, 61, 61, 128) 512
hNormalization)
max_pooling2d_1 (MaxPooling (None, 30, 30, 128) 0
2D)
conv2d_2 (Conv2D) (None, 28, 28, 256) 295168
batch_normalization_2 (Batc (None, 28, 28, 256) 1024
hNormalization)
max_pooling2d_2 (MaxPooling (None, 14, 14, 256) 0
2D)
conv2d_3 (Conv2D) (None, 12, 12, 128) 295040
batch_normalization_3 (Batc (None, 12, 12, 128) 512
hNormalization)
max_pooling2d_3 (MaxPooling (None, 6, 6, 128) 0
2D)
conv2d_4 (Conv2D) (None, 2, 2, 64) 204864
batch_normalization_4 (Batc (None, 2, 2, 64) 256
hNormalization)
max_pooling2d_4 (MaxPooling (None, 1, 1, 64) 0
2D)
flatten (Flatten) (None, 64) 0
dense (Dense) (None, 256) 16640
batch_normalization_5 (Batc (None, 256) 1024
hNormalization)
dropout (Dropout) (None, 256) 0
dense_1 (Dense) (None, 12) 3084
=================================================================
Total params: 894,028
Trainable params: 892,236
Non-trainable params: 1,792
_________________________________________________________________
Model8 Architecture Visualization
#visualize model4 architecture
plot_model(model8, to_file='model8_plot.png', show_shapes=True, show_layer_names=True)
#visualize model8 architecture using visualkeras
visualkeras.layered_view(model8, legend=True)
Early Stopping
Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 100 epochs and validation_split as 10%
history8=model8.fit(X_train,
y_train_onehot,
epochs=100,
batch_size=32,validation_split=0.10,callbacks=[es, mc])
Epoch 1/100 121/121 [==============================] - ETA: 0s - loss: 1.8790 - accuracy: 0.4396 Epoch 1: val_accuracy improved from -inf to 0.12150, saving model to best_model.h5 121/121 [==============================] - 16s 55ms/step - loss: 1.8790 - accuracy: 0.4396 - val_loss: 7.6993 - val_accuracy: 0.1215 Epoch 2/100 120/121 [============================>.] - ETA: 0s - loss: 1.0419 - accuracy: 0.6690 Epoch 2: val_accuracy did not improve from 0.12150 121/121 [==============================] - 6s 49ms/step - loss: 1.0428 - accuracy: 0.6686 - val_loss: 13.5207 - val_accuracy: 0.1215 Epoch 3/100 121/121 [==============================] - ETA: 0s - loss: 0.7727 - accuracy: 0.7466 Epoch 3: val_accuracy improved from 0.12150 to 0.13551, saving model to best_model.h5 121/121 [==============================] - 6s 49ms/step - loss: 0.7727 - accuracy: 0.7466 - val_loss: 11.2206 - val_accuracy: 0.1355 Epoch 4/100 120/121 [============================>.] - ETA: 0s - loss: 0.5631 - accuracy: 0.8055 Epoch 4: val_accuracy improved from 0.13551 to 0.15888, saving model to best_model.h5 121/121 [==============================] - 6s 50ms/step - loss: 0.5654 - accuracy: 0.8050 - val_loss: 7.3115 - val_accuracy: 0.1589 Epoch 5/100 120/121 [============================>.] - ETA: 0s - loss: 0.4607 - accuracy: 0.8375 Epoch 5: val_accuracy improved from 0.15888 to 0.47664, saving model to best_model.h5 121/121 [==============================] - 6s 48ms/step - loss: 0.4605 - accuracy: 0.8378 - val_loss: 1.8527 - val_accuracy: 0.4766 Epoch 6/100 120/121 [============================>.] - ETA: 0s - loss: 0.4055 - accuracy: 0.8552 Epoch 6: val_accuracy improved from 0.47664 to 0.68692, saving model to best_model.h5 121/121 [==============================] - 6s 49ms/step - loss: 0.4069 - accuracy: 0.8547 - val_loss: 1.0092 - val_accuracy: 0.6869 Epoch 7/100 120/121 [============================>.] - ETA: 0s - loss: 0.3637 - accuracy: 0.8711 Epoch 7: val_accuracy improved from 0.68692 to 0.84579, saving model to best_model.h5 121/121 [==============================] - 6s 48ms/step - loss: 0.3633 - accuracy: 0.8713 - val_loss: 0.4853 - val_accuracy: 0.8458 Epoch 8/100 121/121 [==============================] - ETA: 0s - loss: 0.2869 - accuracy: 0.8989 Epoch 8: val_accuracy did not improve from 0.84579 121/121 [==============================] - 6s 47ms/step - loss: 0.2869 - accuracy: 0.8989 - val_loss: 1.3040 - val_accuracy: 0.6145 Epoch 9/100 120/121 [============================>.] - ETA: 0s - loss: 0.2537 - accuracy: 0.9073 Epoch 9: val_accuracy did not improve from 0.84579 121/121 [==============================] - 6s 49ms/step - loss: 0.2535 - accuracy: 0.9075 - val_loss: 0.4965 - val_accuracy: 0.8178 Epoch 10/100 120/121 [============================>.] - ETA: 0s - loss: 0.2117 - accuracy: 0.9229 Epoch 10: val_accuracy improved from 0.84579 to 0.86916, saving model to best_model.h5 121/121 [==============================] - 6s 48ms/step - loss: 0.2137 - accuracy: 0.9218 - val_loss: 0.3837 - val_accuracy: 0.8692 Epoch 11/100 121/121 [==============================] - ETA: 0s - loss: 0.2083 - accuracy: 0.9207 Epoch 11: val_accuracy did not improve from 0.86916 121/121 [==============================] - 6s 50ms/step - loss: 0.2083 - accuracy: 0.9207 - val_loss: 0.4942 - val_accuracy: 0.8318 Epoch 12/100 121/121 [==============================] - ETA: 0s - loss: 0.1778 - accuracy: 0.9350 Epoch 12: val_accuracy did not improve from 0.86916 121/121 [==============================] - 6s 47ms/step - loss: 0.1778 - accuracy: 0.9350 - val_loss: 3.1609 - val_accuracy: 0.4369 Epoch 13/100 120/121 [============================>.] - ETA: 0s - loss: 0.1908 - accuracy: 0.9292 Epoch 13: val_accuracy did not improve from 0.86916 121/121 [==============================] - 6s 48ms/step - loss: 0.1918 - accuracy: 0.9288 - val_loss: 0.4749 - val_accuracy: 0.8621 Epoch 14/100 121/121 [==============================] - ETA: 0s - loss: 0.1592 - accuracy: 0.9436 Epoch 14: val_accuracy did not improve from 0.86916 121/121 [==============================] - 6s 50ms/step - loss: 0.1592 - accuracy: 0.9436 - val_loss: 0.9530 - val_accuracy: 0.7290 Epoch 15/100 120/121 [============================>.] - ETA: 0s - loss: 0.1096 - accuracy: 0.9622 Epoch 15: val_accuracy did not improve from 0.86916 121/121 [==============================] - 6s 47ms/step - loss: 0.1107 - accuracy: 0.9618 - val_loss: 0.4489 - val_accuracy: 0.8621 Epoch 16/100 120/121 [============================>.] - ETA: 0s - loss: 0.1455 - accuracy: 0.9461 Epoch 16: val_accuracy did not improve from 0.86916 121/121 [==============================] - 6s 48ms/step - loss: 0.1462 - accuracy: 0.9459 - val_loss: 1.3297 - val_accuracy: 0.6799 Epoch 17/100 121/121 [==============================] - ETA: 0s - loss: 0.1320 - accuracy: 0.9527 Epoch 17: val_accuracy did not improve from 0.86916 121/121 [==============================] - 6s 46ms/step - loss: 0.1320 - accuracy: 0.9527 - val_loss: 0.4943 - val_accuracy: 0.8411 Epoch 18/100 120/121 [============================>.] - ETA: 0s - loss: 0.0829 - accuracy: 0.9703 Epoch 18: val_accuracy did not improve from 0.86916 121/121 [==============================] - 6s 49ms/step - loss: 0.0833 - accuracy: 0.9701 - val_loss: 0.5585 - val_accuracy: 0.8458 Epoch 19/100 121/121 [==============================] - ETA: 0s - loss: 0.0833 - accuracy: 0.9711 Epoch 19: val_accuracy did not improve from 0.86916 121/121 [==============================] - 6s 46ms/step - loss: 0.0833 - accuracy: 0.9711 - val_loss: 1.4980 - val_accuracy: 0.6893 Epoch 20/100 121/121 [==============================] - ETA: 0s - loss: 0.1280 - accuracy: 0.9537 Epoch 20: val_accuracy did not improve from 0.86916 121/121 [==============================] - 6s 47ms/step - loss: 0.1280 - accuracy: 0.9537 - val_loss: 3.8885 - val_accuracy: 0.4486 Epoch 20: early stopping
Training stopped at epoch 20. The highest validation accuracy was 0.8916 with a validation loss of 0.3837.
Model8 Accuracy vs Epoch curve
plt.plot(history8.history['accuracy'])
plt.plot(history8.history['val_accuracy'])
plt.title('model8 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
The model overfits.
Model8 Evaluation of model model.evaluate using the Test data
# Evaluate the model on the test data
score8 = model8.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score8[0])
print('Test accuracy:', score8[1])
Test loss: 3.9417619705200195 Test accuracy: 0.43157893419265747
# Test Prediction
y_pred = model8.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)
15/15 [==============================] - 0s 15ms/step
# Test Accuracy
accuracyScore8 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore8)
0.43157894736842106
Classification Report
# Compute the classification report
cr8 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)
print('Classification report:\n',cr8)
Classification report:
precision recall f1-score support
Small-flowered Cranesbill 0.58 0.80 0.67 50
Fat Hen 1.00 0.08 0.15 48
Shepherds Purse 0.50 0.26 0.34 23
Common wheat 0.65 0.50 0.56 22
Common Chickweed 1.00 0.03 0.06 61
Charlock 0.76 0.72 0.74 39
Cleavers 0.90 0.31 0.46 29
Scentless Mayweed 0.25 0.96 0.40 52
Sugar beet 0.50 0.05 0.10 38
Maize 0.23 0.50 0.31 22
Black-grass 0.80 0.15 0.26 26
Loose Silky-bent 0.58 0.58 0.58 65
accuracy 0.43 475
macro avg 0.64 0.41 0.39 475
weighted avg 0.66 0.43 0.38 475
Overall the accuracy is low at 0.43. We have developed better models earlier.
Scoresdf = Scoresdf.append({'model':'Model8','accuracy':accuracyScore8,'layers':'20','total parameters':'894,028', 'neurons':'256', 'optimizer':'Adam'},ignore_index=True)
Confusion Matrix
# Compute Confusion matrix
cm8 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
sns.heatmap(cm8, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()
From the confusion matrix we can observe that Model8 is unable to predict many classes correctly at all. Overall, this is not a good model.
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
This will be our final model trying to improve our previous models. Again, we will modify the number of layers, filter size and neurons.
Model Implementation
# Intializing a sequential model
model9 = Sequential()
# Adding first conv layer with 64 filters and kernel size 3x3 , padding 'same' provides the output size same as the input size
# Input_shape denotes input image dimension of MNIST images
model9.add(Conv2D(64, (3, 3), activation='relu', padding="same", input_shape=input_shape))
model9.add(Conv2D(64, (3, 3), activation='relu', padding="same"))
# Adding max pooling to reduce the size of output of first conv layer
model9.add(MaxPooling2D((2, 2), padding = 'same'))
model9.add(Conv2D(32, (3, 3), activation='relu', padding="same"))
model9.add(Conv2D(32, (3, 3), activation='relu', padding="same"))
model9.add(MaxPooling2D((2, 2), padding = 'same'))
model9.add(Conv2D(32, (3, 3), activation='relu', padding="same"))
model9.add(Conv2D(32, (3, 3), activation='relu', padding="same"))
model9.add(MaxPooling2D((2, 2), padding = 'same'))
# flattening the output of the conv layer after max pooling to make it ready for creating dense connections
model9.add(Flatten())
# Adding a fully connected dense layer with 100 neurons
model9.add(Dense(128, activation='relu'))
model9.add(Dropout(0.25))
model9.add(Dense(64, activation='relu'))
# Adding the output layer with 10 neurons and activation functions as softmax since this is a multi-class classification problem
model9.add(Dense(num_classes, activation='softmax'))
# Using Adam Optimizer
opt = Adam()
# Compile model
model9.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model9.summary()
Model: "sequential_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_6 (Conv2D) (None, 128, 128, 64) 1792
conv2d_7 (Conv2D) (None, 128, 128, 64) 36928
max_pooling2d_3 (MaxPooling (None, 64, 64, 64) 0
2D)
conv2d_8 (Conv2D) (None, 64, 64, 32) 18464
conv2d_9 (Conv2D) (None, 64, 64, 32) 9248
max_pooling2d_4 (MaxPooling (None, 32, 32, 32) 0
2D)
conv2d_10 (Conv2D) (None, 32, 32, 32) 9248
conv2d_11 (Conv2D) (None, 32, 32, 32) 9248
max_pooling2d_5 (MaxPooling (None, 16, 16, 32) 0
2D)
flatten_1 (Flatten) (None, 8192) 0
dense_3 (Dense) (None, 128) 1048704
dropout_1 (Dropout) (None, 128) 0
dense_4 (Dense) (None, 64) 8256
dense_5 (Dense) (None, 12) 780
=================================================================
Total params: 1,142,668
Trainable params: 1,142,668
Non-trainable params: 0
_________________________________________________________________
Model9 Architecture Visualization
#visualize model9 architecture
plot_model(model9, to_file='model9_plot.png', show_shapes=True, show_layer_names=True)
#visualize model9 architecture using visualkeras
visualkeras.layered_view(model9, legend=True)
Early Stopping
Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 100 epochs and validation_split as 10%
history9=model9.fit(X_train,
y_train_onehot,
epochs=100,
batch_size=32,validation_split=0.10,callbacks=[es, mc])
Epoch 1/100 121/121 [==============================] - ETA: 0s - loss: 2.3441 - accuracy: 0.1869 Epoch 1: val_accuracy improved from -inf to 0.33645, saving model to best_model.h5 121/121 [==============================] - 15s 71ms/step - loss: 2.3441 - accuracy: 0.1869 - val_loss: 1.9613 - val_accuracy: 0.3364 Epoch 2/100 120/121 [============================>.] - ETA: 0s - loss: 1.6789 - accuracy: 0.4146 Epoch 2: val_accuracy improved from 0.33645 to 0.42991, saving model to best_model.h5 121/121 [==============================] - 6s 53ms/step - loss: 1.6793 - accuracy: 0.4143 - val_loss: 1.5914 - val_accuracy: 0.4299 Epoch 3/100 120/121 [============================>.] - ETA: 0s - loss: 1.4057 - accuracy: 0.5120 Epoch 3: val_accuracy improved from 0.42991 to 0.57009, saving model to best_model.h5 121/121 [==============================] - 6s 51ms/step - loss: 1.4049 - accuracy: 0.5123 - val_loss: 1.2518 - val_accuracy: 0.5701 Epoch 4/100 121/121 [==============================] - ETA: 0s - loss: 1.1737 - accuracy: 0.5976 Epoch 4: val_accuracy improved from 0.57009 to 0.58879, saving model to best_model.h5 121/121 [==============================] - 6s 53ms/step - loss: 1.1737 - accuracy: 0.5976 - val_loss: 1.1742 - val_accuracy: 0.5888 Epoch 5/100 120/121 [============================>.] - ETA: 0s - loss: 0.9768 - accuracy: 0.6596 Epoch 5: val_accuracy improved from 0.58879 to 0.63785, saving model to best_model.h5 121/121 [==============================] - 6s 51ms/step - loss: 0.9767 - accuracy: 0.6597 - val_loss: 1.0805 - val_accuracy: 0.6379 Epoch 6/100 120/121 [============================>.] - ETA: 0s - loss: 0.8448 - accuracy: 0.7049 Epoch 6: val_accuracy improved from 0.63785 to 0.70561, saving model to best_model.h5 121/121 [==============================] - 6s 52ms/step - loss: 0.8442 - accuracy: 0.7052 - val_loss: 0.8990 - val_accuracy: 0.7056 Epoch 7/100 121/121 [==============================] - ETA: 0s - loss: 0.7357 - accuracy: 0.7385 Epoch 7: val_accuracy improved from 0.70561 to 0.70794, saving model to best_model.h5 121/121 [==============================] - 6s 50ms/step - loss: 0.7357 - accuracy: 0.7385 - val_loss: 0.8873 - val_accuracy: 0.7079 Epoch 8/100 120/121 [============================>.] - ETA: 0s - loss: 0.6930 - accuracy: 0.7492 Epoch 8: val_accuracy did not improve from 0.70794 121/121 [==============================] - 7s 55ms/step - loss: 0.6923 - accuracy: 0.7494 - val_loss: 0.8996 - val_accuracy: 0.7056 Epoch 9/100 120/121 [============================>.] - ETA: 0s - loss: 0.5849 - accuracy: 0.7870 Epoch 9: val_accuracy did not improve from 0.70794 121/121 [==============================] - 6s 54ms/step - loss: 0.5846 - accuracy: 0.7868 - val_loss: 0.9644 - val_accuracy: 0.7079 Epoch 10/100 120/121 [============================>.] - ETA: 0s - loss: 0.5359 - accuracy: 0.8005 Epoch 10: val_accuracy improved from 0.70794 to 0.73832, saving model to best_model.h5 121/121 [==============================] - 6s 53ms/step - loss: 0.5360 - accuracy: 0.8006 - val_loss: 0.8499 - val_accuracy: 0.7383 Epoch 11/100 120/121 [============================>.] - ETA: 0s - loss: 0.4818 - accuracy: 0.8193 Epoch 11: val_accuracy did not improve from 0.73832 121/121 [==============================] - 6s 50ms/step - loss: 0.4820 - accuracy: 0.8191 - val_loss: 0.8597 - val_accuracy: 0.7383 Epoch 12/100 120/121 [============================>.] - ETA: 0s - loss: 0.4475 - accuracy: 0.8406 Epoch 12: val_accuracy improved from 0.73832 to 0.74766, saving model to best_model.h5 121/121 [==============================] - 6s 51ms/step - loss: 0.4473 - accuracy: 0.8407 - val_loss: 0.9585 - val_accuracy: 0.7477 Epoch 13/100 120/121 [============================>.] - ETA: 0s - loss: 0.3718 - accuracy: 0.8628 Epoch 13: val_accuracy did not improve from 0.74766 121/121 [==============================] - 6s 50ms/step - loss: 0.3718 - accuracy: 0.8625 - val_loss: 0.9823 - val_accuracy: 0.7336 Epoch 14/100 121/121 [==============================] - ETA: 0s - loss: 0.3761 - accuracy: 0.8646 Epoch 14: val_accuracy did not improve from 0.74766 121/121 [==============================] - 6s 52ms/step - loss: 0.3761 - accuracy: 0.8646 - val_loss: 1.1009 - val_accuracy: 0.7196 Epoch 15/100 121/121 [==============================] - ETA: 0s - loss: 0.3230 - accuracy: 0.8763 Epoch 15: val_accuracy did not improve from 0.74766 121/121 [==============================] - 6s 51ms/step - loss: 0.3230 - accuracy: 0.8763 - val_loss: 1.0405 - val_accuracy: 0.7453 Epoch 16/100 121/121 [==============================] - ETA: 0s - loss: 0.3398 - accuracy: 0.8692 Epoch 16: val_accuracy improved from 0.74766 to 0.75701, saving model to best_model.h5 121/121 [==============================] - 6s 51ms/step - loss: 0.3398 - accuracy: 0.8692 - val_loss: 0.9032 - val_accuracy: 0.7570 Epoch 17/100 120/121 [============================>.] - ETA: 0s - loss: 0.3130 - accuracy: 0.8878 Epoch 17: val_accuracy did not improve from 0.75701 121/121 [==============================] - 6s 51ms/step - loss: 0.3133 - accuracy: 0.8877 - val_loss: 0.9715 - val_accuracy: 0.7477 Epoch 18/100 120/121 [============================>.] - ETA: 0s - loss: 0.2706 - accuracy: 0.9003 Epoch 18: val_accuracy improved from 0.75701 to 0.77103, saving model to best_model.h5 121/121 [==============================] - 6s 50ms/step - loss: 0.2708 - accuracy: 0.8999 - val_loss: 0.9955 - val_accuracy: 0.7710 Epoch 19/100 121/121 [==============================] - ETA: 0s - loss: 0.2637 - accuracy: 0.9038 Epoch 19: val_accuracy did not improve from 0.77103 121/121 [==============================] - 6s 51ms/step - loss: 0.2637 - accuracy: 0.9038 - val_loss: 1.1545 - val_accuracy: 0.7103 Epoch 20/100 120/121 [============================>.] - ETA: 0s - loss: 0.2262 - accuracy: 0.9130 Epoch 20: val_accuracy improved from 0.77103 to 0.79907, saving model to best_model.h5 121/121 [==============================] - 6s 50ms/step - loss: 0.2273 - accuracy: 0.9129 - val_loss: 0.9187 - val_accuracy: 0.7991 Epoch 20: early stopping
Training stopped at epoch 20. The highest validation accuracy was 0.79907 with a validation loss of 1.145.
Model9 Accuracy vs Epoch curve
plt.plot(history9.history['accuracy'])
plt.plot(history9.history['val_accuracy'])
plt.title('model9 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
The model overfits.
Model9 Evaluation of model model.evaluate using the Test data
# Evaluate the model on the test data
score9 = model9.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score8[0])
print('Test accuracy:', score8[1])
Test loss: 3.9417619705200195 Test accuracy: 0.43157893419265747
# Test Prediction
y_pred = model9.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)
15/15 [==============================] - 0s 16ms/step
# Test Accuracy
accuracyScore9 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore8)
0.43157894736842106
Classification Report
# Compute the classification report
cr9 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)
print('Classification report:\n',cr9)
Classification report:
precision recall f1-score support
Small-flowered Cranesbill 0.93 0.86 0.90 50
Fat Hen 0.68 0.88 0.76 48
Shepherds Purse 0.62 0.43 0.51 23
Common wheat 0.72 0.82 0.77 22
Common Chickweed 0.85 0.87 0.86 61
Charlock 0.86 0.92 0.89 39
Cleavers 0.77 0.93 0.84 29
Scentless Mayweed 0.74 0.75 0.74 52
Sugar beet 0.92 0.61 0.73 38
Maize 0.67 0.73 0.70 22
Black-grass 0.50 0.15 0.24 26
Loose Silky-bent 0.75 0.89 0.82 65
accuracy 0.78 475
macro avg 0.75 0.74 0.73 475
weighted avg 0.77 0.78 0.76 475
Model9 with four convolutional layers indicates that the overall performance is good, with an accuracy of 78%.
Scoresdf = Scoresdf.append({'model':'Model9','accuracy':accuracyScore9,'layers':'14','total parameters':'1,142,668', 'neurons':'64', 'optimizer':'Adam'},ignore_index=True)
Confusion Matrix
# Compute Confusion matrix
cm9 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
sns.heatmap(cm9, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()
Model9 performs well but not as good as Model4.
We tried to improve our model by implementing 9 models. One approach to implement a model is by changing the hyperparameters like optimizer, number of neurons, number of filters, adding convolution layers, batchnormalizers, changing the dropout rate, etc.
We performed these to improve our model's performance:
Architecture Modifications: Modifying the architecture of the CNN by adding or removing layers, changing the number of filters or neurons in each layer, or using different activation functions can improve performance. We implemented diverse models this way.
Hyperparameter Tuning: Optimizing the hyperparameters of the model, such as learning rate, batch size, and optimizer, can improve performance. This can be done using techniques such as grid search or random search. We changed the optimizer (Adam, SGD).
Regularization: Regularization techniques such as dropout and L2 regularization can prevent overfitting and improve generalization performance. We performed dropout.
Batch Normalization: Batch normalization is a technique that normalizes the inputs of each layer to prevent internal covariate shift. This can improve performance and reduce the number of epochs needed for training. We used batch Normalization.
Early Stopping: Early stopping is a technique that stops training the model when the validation loss stops improving. This prevents overfitting and improves generalization performance. We used Early stopping.
We didn't implement:
Data Augmentation: Increasing the amount of data available for training the model can improve performance. Data augmentation techniques such as flipping, rotating, scaling, and cropping images can create new variations of the same images and help the model to generalize better. However, this may require computational power so it may be expensive to perform.
Transfer Learning: Pretrained models can be used as a starting point for training a new model. Transfer learning involves using the weights of a model trained on a large dataset, such as ImageNet, to initialize the weights of a new model. This can speed up training and improve performance.
Ensembling: Ensemble methods involve combining the predictions of multiple models to improve performance. This can be done by training multiple CNN models with different hyperparameters or architectures and combining their predictions.
From Model Improvement we came up with 9 different models which are compared below:
Scoresdf
| model | accuracy | layers | total parameters | neurons | optimizer | |
|---|---|---|---|---|---|---|
| 0 | Model1 | 0.732632 | 8 | 7,393,868 | 128 | Adam |
| 1 | Model2 | 0.791579 | 10 | 3,306,188 | 128 | Adam |
| 2 | Model4 | 0.833684 | 17 | 834,252 | 128 | Adam |
| 3 | Model3 | 0.757895 | 10 | 3,306,188 | 128 | SGD |
| 4 | Model5 | 0.825263 | 20 | 821,324 | 256 | Adam |
| 5 | Model6 | 0.562105 | 17 | 26,200,716 | 256 | Adam |
| 6 | Model7 | 0.244211 | 20 | 9,733,804 | 512 | Adam |
| 7 | Model8 | 0.431579 | 20 | 894,028 | 256 | Adam |
| 8 | Model9 | 0.776842 | 14 | 1,142,668 | 64 | Adam |
The model that performed best was Model4 with an overall accuracy of 0.83.
Model4 has 17 layers and trained 834,52 parameters. The fully connected layer had 128 neurons and the optimizer run was Adam.
The precision scores range from 0.48 to 0.95, indicating that the model's ability to correctly identify positive instances varies by class. For example, the model has high precision in identifying Charlock and Cleavers, while it has relatively lower precision in identifying Black-grass and Shepherds Purse.
The recall scores range from 0.50 to 0.97, indicating that the model's ability to correctly identify positive instances also varies by class. For example, the model has high recall in identifying Common Chickweed and Charlock, while it has relatively lower recall in identifying Black-grass and Shepherds Purse.
The F1-scores are a harmonic mean of precision and recall and range from 0.49 to 0.92. The F1-score provides a balance between precision and recall, indicating the overall performance of the model for each class.
The best performing model obtained has 4 Conv2D layers, 128 filters per layer, and 2 Dense layers with 256 neurons each. This model achieved an accuracy of 83% on the test set, which is a significant improvement from the baseline model.
We could further try to improve the model's performance by trying mostly:
Increase the size of the training dataset by collecting more images or using data augmentation techniques to generate new images from the existing ones.
Try using pre-trained models such as VGG, ResNet or Inception and fine-tune them on your dataset.
Continue to experiment with different hyperparameters such as batch size, number of epochs, and regularization techniques to find the best combination for your model.
We should notice that building a good performing CNN model requires a lot of experimentation and fine-tuning.
Additionally keep in mind:
Overall, our model performs well but it needs to be fine tuned for the classes that are most difficult to classify, in our case, Black grass.